Parameter uncertainity in least squares optimization: rescaling Hessian

Question

Given a least squares optimization problem of the form:

$$ C(\lambda) = \sum_i ||y_i - f(x_i, \lambda)||^2$$

I have found in multiple questions/answers (e.g. here) that an estimate for the covariance of the parameters can be computed from the inverse rescaled Hessian at the minimum point:

$$ \mathrm{cov}(\hat\lambda) = \hat H^{-1} \hat\sigma_r^2 = \hat H^{-1} \frac{\sum_i ||y_i - f(x_i, \hat\lambda)||^2}{N_{DOF}} $$

While I understand why the covariance is related to the inverse Hessian (Fisher information), I haven't found anywhere a demonstration or explanation for the $\hat\sigma_r^2$ term, although it appears reasonable to me on intuitive grounds.

Could anybody explain the need for the rescaling by the residual variance and/or provide a reference?

Answer 1

One can possibly look at the approximation for the covariance as a "rescaled inverse Hessian", but it kind of hides the simple deduction.

In principle it's the stopped series expansion of the term that is minimized. So, if one stops at the second term, the expression is in the simple 1D case around the (minimum) value of a: $$f(x) = f(a) + f^\prime(a)(x-a) + \frac{1}{2}f^{\prime\prime}(a)(x-a)^2$$ If you change to the vectorized version, the derivatives are replaced by their respective vectorized parts.

At the minimum the slope is zero, which means the first derivative -- the Jacobian -- is zero. E.g. also https://en.wikipedia.org/wiki/Taylor_expansions_for_the_moments_of_functions_of_random_variables So, now you can identify the remaining part with the expression of the approximation at the minimum.

The actual scaling factor of the variance is a bit more complicated, why the number of degrees of freedom is a good choice. The logic behind this is that the term should neither be under-, nor overestimated. Sometimes this is described as "unbiased", but in my understanding this implies a dependency on the value itself. So, in my understanding it's more like a correction that can be verified by using analytical examples.