7
$\begingroup$

Is there a name for approximating the Hessian as the outer product of the gradient with itself?

If one is approximating the Hessian of the log-loss, then the outer product of the gradient with itself is the Fisher information matrix. What about in general?

I'm looking at this in the context of explaining what the so-called Gauss-Newton matrix assumes (Schraudolph, N. N. (2002). Fast curvature matrix-vector products for second-order gradient descent. Neural Computation, 14(7), 1723–38.). We have an input vector, followed by a linear transformation, followed by a nonlinear loss function. The Hessian of the linear transformation (A) is approximated as an outer-product of gradients. The Hessian of the negative log-loss function (B) is assumed to be positive semidefinite. My question is what is the assumption A called?

|cite|improve this question
$\endgroup$
6
$\begingroup$

The expected value of the outer product of the gradient of the log-likelihood is the "information matrix", or "Fisher information" irrespective of whether we use it instead of the negative of the Hessian or not, see this post. It is also the "variance of the score".

The relation that permits us to use the outer product of the gradient instead of the negative of the Hessian, is called the Information Matrix Equality, and it is valid under the assumption of correct specification (this is important but usually goes unmentioned), as well as some regularity conditions that permit the interchange of integration and differentiation.

Perhaps this could be useful also.

Note: In many corners it is just said "outer product of the gradient" without adding "with itself".

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thanks, that's great for building intuition about what this approximation assumes. Do you know if this approximation is made in general cases not involving a loss function? $\endgroup$ – Neil G Oct 31 '14 at 6:28
  • $\begingroup$ The approximation is related to the asymptotic variance of the maximum likelihood estimator. Its full expression is $$Avar(MLE) = [E(H)]^{-1}E(SS')[E(H)]^{-1}$$ and due to the relation, $E(SS')=-[E(H)]$ it simplifies to $-[E(H)]^{-1}$ or $[E(SS')]^{-1}$. $\endgroup$ – Alecos Papadopoulos Oct 31 '14 at 6:34
  • $\begingroup$ Thanks, I'm not sure I understand, but I've edited my question to try to be more clear. $\endgroup$ – Neil G Oct 31 '14 at 6:38
4
$\begingroup$

The outer product of gradient estimator for the covariance matrix of maximum likelihood estimates is also known as the BHHH estimator, because it was proposed by Berndt, Hall, Hall and Hausman in this paper:

Berndt, E.K., Hall, B.H., Hall, R.E. and Hausman, J.A. (1974). 
"Estimation and Inference in Nonlinear Structural Models". 
Annals of Economic and Social Measurement, 3, pp. 653-665.

In the discussion around equation (3.8) of the paper you may get further details justifying the use of this expression.

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.