Relation between Covariance matrix and Jacobian in Nonlinear Least Squares

Question

I saw that CovB = inv(J'*J)*MSE in a MATLAB documentation here at http://www.mathworks.com/help/stats/nlinfit.html However, I cant find any sources for the relationship . I believe in it but I need to find a source to refer it. I have looked but I cant find. It would be great if anyone can help me a bit.

Answer 1

This is based on the standard approximation to the Hessian of a nonlinear least squares problem used by Gauss-Newton and Levenberg-Marquardt algorithms.

Consider the nonlinear least squares problem: minimize $1/2r(x)^Tr(x)$. Let $J$ = Jacobian of r(x). The Hessian of the objective = $J^TJ +$ higher order terms. The Gauss-Newton or Levenberg-Marquardt approximation is to ignore the higher order terms, and approximate the Hessian as $J^TJ$. This approximation for the Hessian is what is used in the formula CovB = inv(J'*J)*MSE in MATLAB's nlinfit.

The higher order terms are close to zero at the solution if the residuals r(x) are close to zero. If the residuals are large at the solution, the approximation may be very inaccurate. See the first 7 slides of https://www8.cs.umu.se/kurser/5DA001/HT07/lectures/lsq-handouts.pdf . There is also mention, with less detail provided, at https://en.wikipedia.org/wiki/Gauss%E2%80%93Newton_algorithm.