4
$\begingroup$

I am trying to understand the intuition behind the "information matrix equality" condition in the Maximum Likelihood context (perhaps this is the only context?): $$ -E[H(\theta)] = E[s(\theta) s(\theta)'] \quad\quad\quad\quad\quad\quad\quad(1) $$ where $\theta$ is the vector of parameters, $H$ is the Hessian ("curvature of the log-likelihood w.r.t. $\theta$") and $s$ is the score ("gradient of the log-likelihood").

I understand that the curvature of the log-likelihood tells us the amount/precision of "information" that the sample provides us about $\theta$. Hence, I can appreciate the "information" part.

However, I fail to understand the intuition (the math is clear) for why $(1)$ holds. Why would the curvature be approximated by the "square" of the gradient? If the Hessian is a square matrix of second-derivatives, then what is formulation (1) telling us about the off-diagonals - it seems like they would be zero or the same as the main diagonal? It reminds me of the assumption from OLS that the variance of the errors is homoskedastic ($E[X'uu'X]=\sigma^2 I$) - this allows us to go from the robust ("sandwich") form for the variance-covariance matrix to the form $\sigma^2(X'X)^-1$. It seems like here we are making a similar point about the off-diagonals, but I can't work it out.

Pages 10-11 here and pages 7-8 here are helpful but don't really seem to hammer home why this result works, in an intuitive manner. Thank you for any help and Happy New Year!

$\endgroup$
  • 2
    $\begingroup$ Note: it is not the "square" of the gradient (or score) but its variance-covariance matrix. Since the expectation of the score is zero. The information thus tells you how much the gradient (vector) varies. The more it varies, the faster the likelihood drops away from its maximum and the closest the maximum gets to the true value. $\endgroup$ – Xi'an Jan 2 '15 at 20:34
  • $\begingroup$ What is the meaning of the subscript "$i$"? It doesn't seem like it could be a component index, because $H$ is a matrix. It might be worth noting that the components of $H$ are not actually "curvatures," even though they are often described by that term. $\endgroup$ – whuber Jan 2 '15 at 23:02
  • $\begingroup$ Xi'an - that was lovely and concise, thank you. whuber<> - I will edit to remove the "i" subscript. Could you expand on why they aren't "curvatures"? They are the second derivatives of a function - is this not the curvature? $\endgroup$ – singlepeaked Jan 3 '15 at 12:28
  • $\begingroup$ The curvature is the radius of the circle that best fits within a curve. It is inversely proportional to the absolute value of the second derivative at that point. $\endgroup$ – Placidia Jan 3 '15 at 16:28
  • $\begingroup$ The second derivative of a function is almost never an actual curvature. In this case, the right hand side of your equation (1) gives a Riemannian metric. From it a (Riemannian) curvature can be derived by taking additional derivatives; this curvature generally is a tensor requiring four subscripts. Although there are obvious relationships between second derivatives and curvatures, they depend on exactly how the manifold is parameterized (which is an arbitrary choice often without intrinsic meaning). The curvature, being independent of the parameterization, has intrinsic meaning. $\endgroup$ – whuber Jan 3 '15 at 16:28
1
$\begingroup$

I have always found this result to be counter-intuitive as well. In my case, this is due to a tendency to confuse the score function and the maximum likelihood estimator. (Oh, the shame!). In fact, they sort of pull in opposite directions.

Consider the one parameter case. In large samples, the log likelihood tends towards a quadratic function open below. The second derivative will be large and negative when the quadratic is tight around the maximum and the curvature (reciprocal of the second derivative) will be small. Intuitively, it makes sense that the variance of the scaled MLE should be the reciprocal of the Fisher information (LHS of the identity). And it is.

But the RHS of the identify is not the MLE. On the contrary. The score function depends on both the sample values and the true parameter. If we knew the true value of the parameter and drew many samples, $s(x,\theta)$ will vary about 0. The RHS is the variance of that quantity.

In the absurd case where the sample is of size 1 and the distribution is a normal with known variance, the RHS is the variance of $$\frac{x-\mu}{\sigma^2}$$

If I increase the sample to $n$ IID values, I am looking at the variance of $$\sum \frac{x_i - \mu}{\sigma^2}$$

The bigger my sample, the larger the variance, while of course, the variance of the MLE is shrinking. Meanwhile the Hessian is growing more negative, as I add more terms. Think of the score function as a random walk, whose variance grows with the sample size. At the same time, as the sample size grows, my knowledge of $\theta$ is also growing. But knowledge of $\theta$ is captured in the variance of the MLE, which is the reciprocal of the Information matrix (the LHS).

In the normal case, it is trivial to show that the RHS equals the LHS. From there, it makes sense that the two sides should tend to equality when the samples get large. I still find it amazing that the equality is not asymptotic, but that it holds semper ubique. However, you have seen the proof. I'm just trying to get at the intuition.

Later

Still trying to wrap my head around this, I went back to the actual proof of the identity, hoping that it would clarify why the result works. The proof depends totally on the fact that the score function is the derivative of the log likelihood - and happy cancellations occur when evaluating the integrals. Also relevant is the fact that a density function tends to 0 at both plus and minus infinity. There isn't normally a nice connection between the variance of a random variable and its expected derivative with respect to the parameter of the underlying distributional family. It's all part of the magic of logarithms and $$\frac{d \log(f(x))}{dx} = \frac{f'(x)}{f(x)}$$

I don't think your question has an answer. Or rather, the questions should be "why is the log likelihood a good thing?" and "Why is the Hessian called an information matrix?"; and the answer to those questions is the Information identity.

Regarding @singlepeaked 's comparison to the OLS variance, remember that the OLS estimates of the coefficients when $\sigma$ is known and the error is normal is also the maximum likelihood estimator. The var-cov matrix of those estimates is an example of the Information identity at work.

$\endgroup$
  • $\begingroup$ Placidia - thank you for this answer. I now understand that the information equality is saying that the expectation of the information our sample provides us about $\theta$ is just the expected volatility in the gradient of the log-likelihood function - do you think this captures your points? $\endgroup$ – singlepeaked Jan 3 '15 at 12:37
  • $\begingroup$ @singlepeaked Remember that the identity is only true at the true value of the parameter. The takeaway is that the variance of the score function is the inverse of the variance of the MLE. $\endgroup$ – Placidia Jan 3 '15 at 16:25
0
$\begingroup$

I have to say that I am not sure what you are confused about.

"it seems like they would be zero or the same as the main diagonal?"

Not really - expected value of the score is zero but the cross products aren't (the different components of the score vector are not independent) - it is a matrix multiplication.

$\endgroup$
  • $\begingroup$ My confusion is this: (ignoring the minus sign) why, intuitively, can we replace the Hessian with the outer product of the scores? The reason I relate this point to the off-diagonals is because of the analogy to OLS: when we can say our error covariances are zero, the robust var-cov matrix collapses to $\sigma^2 (X'X)^-1$. It seems like something similar is happening here... $\endgroup$ – singlepeaked Jan 3 '15 at 12:32
  • $\begingroup$ Further to this point, I decided to read up on M-Estimation in panel data models. The key use for the information equality result is to simplify the variance-covariance matrix. In panel data methods, this result is only valid when we have a dynamically complete model. Testing for dynamic completeness exploits the fact that it implies zero serial correlation in the residuals. This is very much related to the off-diagonals. $\endgroup$ – singlepeaked Jan 3 '15 at 15:19
  • $\begingroup$ @singlepeaked "ignoring the minus sign" is fatal. That minus sign enters through integration by parts. It's what makes everything work. See my edited answer below. $\endgroup$ – Placidia Jan 3 '15 at 17:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.