Gauss–Newton Residual Curvature

In nonlinear least-squares optimization, the objective function is the sum of squared residuals, f(x) = 0.5 * ||r(x)||^2. The true Hessian matrix of this objective contains two terms: the Gauss–Newton matrix G = J^T J, which is always positive semi-definite and only depends on the first derivatives of the residuals, and a second term that involves the second derivatives of the residuals weighted by the residuals themselves. Separate positive Gauss–Newton curvature from the residual term in the true Hessian.

Experiment: Exploring Curvature Differences

We can observe how the Gauss-Newton approximation differs from the true Hessian using the interactive module.

1. Open the Gauss–Newton Residual Curvature module.
2. Set Point x to 0, Point y to 0, and Damping to 0.
3. Observe that the GN minimum eigenvalue is 1.0, while the True Hessian minimum eigenvalue is -1.0. The Gauss-Newton matrix is positive definite, but the true Hessian is negative definite, indicating a local maximum despite a zero gradient.
4. Set Point x to 0.5, Point y to 0.5, and Damping to 0. The Gauss-Newton matrix becomes singular (minimum eigenvalue 0.0), making the GN step x and GN step y Unavailable.
5. Set Damping to 1 to add regularization. Now, both the damped Gauss-Newton matrix and the damped true Hessian become invertible, and the steps can be computed.