Intuition for the "information matrix equality" result? I am trying to understand the intuition behind the "information matrix equality" condition in the Maximum Likelihood context (perhaps this is the only context?):
$$
-E[H(\theta)] = E[s(\theta) s(\theta)']  \quad\quad\quad\quad\quad\quad\quad(1)
$$
where $\theta$ is the vector of parameters, $H$ is the Hessian ("curvature of the log-likelihood w.r.t. $\theta$") and $s$ is the score ("gradient of the log-likelihood"). 
I understand that the curvature of the log-likelihood tells us the amount/precision of "information" that the sample provides us about $\theta$. Hence, I can appreciate the "information" part. 
However, I fail to understand the intuition (the math is clear) for why $(1)$ holds. Why would the curvature be approximated by the "square" of the gradient? If the Hessian is a square matrix of second-derivatives, then what is formulation (1) telling us about the off-diagonals - it seems like they would be zero or the same as the main diagonal?  It reminds me of the assumption from OLS that the variance of the errors is homoskedastic ($E[X'uu'X]=\sigma^2 I$) - this allows us to go from the robust ("sandwich") form for the variance-covariance matrix to the form $\sigma^2(X'X)^-1$. It seems like here we are making a similar point about the off-diagonals, but I can't work it out. 
Pages 10-11 here and pages 7-8 here are helpful but don't really seem to hammer home why this result works, in an intuitive manner. Thank you for any help and Happy New Year!
 A: I have always found this result to be counter-intuitive as well. In my case, this is due to a tendency to confuse the score function and the maximum likelihood estimator. (Oh, the shame!). In fact, they sort of pull in opposite directions.
Consider the one parameter case. In large samples, the log likelihood tends towards a quadratic function open below. The second derivative will be large and negative when the quadratic is tight around the maximum and the curvature (reciprocal of the second derivative) will be small. Intuitively, it makes sense that the variance of the scaled MLE should be the reciprocal of the Fisher information (LHS of the identity). And it is.
But the RHS of the identify is not the MLE. On the contrary. The score function depends on both the sample values and the true parameter. If we knew the true value of the parameter and drew many samples, $s(x,\theta)$ will vary about 0. The RHS is the variance of that quantity. 
In the absurd case where the sample is of size 1 and the distribution is a normal with known variance, the RHS is the variance of $$\frac{x-\mu}{\sigma^2}$$
If I increase the sample to $n$ IID values, I am looking at the variance of $$\sum \frac{x_i - \mu}{\sigma^2}$$
The bigger my sample, the larger the variance, while of course, the variance of the MLE is shrinking. Meanwhile the Hessian is growing more negative, as I add more terms. Think of the score function as a random walk, whose variance grows with the sample size. At the same time, as the sample size grows, my knowledge of $\theta$ is also growing. But knowledge of $\theta$ is captured in the variance of the MLE, which is the reciprocal of the Information matrix (the LHS).
In the normal case, it is trivial to show that the RHS equals the LHS. From there, it makes sense that the two sides should tend to equality when the samples get large. I still find it amazing that the equality is not asymptotic, but that it holds semper ubique. However, you have seen the proof. I'm just trying to get at the intuition.
Later
Still trying to wrap my head around this, I went back to the actual proof of the identity, hoping that it would clarify why the result works. The proof depends totally on the fact that the score function is the derivative of the log likelihood - and happy cancellations occur when evaluating the integrals. Also relevant is the fact that a density function tends to 0 at both plus and minus infinity. There isn't normally a nice connection between the variance of a random variable and its expected derivative with respect to the parameter of the underlying distributional family. It's all part of the magic of logarithms and $$\frac{d \log(f(x))}{dx} = \frac{f'(x)}{f(x)}$$
I don't think your question has an answer. Or rather, the questions should be "why is the log likelihood a good thing?" and "Why is the Hessian called an information matrix?"; and the answer to those questions is the Information identity.
Regarding @singlepeaked 's comparison to the OLS variance, remember that the OLS estimates of the coefficients when $\sigma$ is known and the error is normal is also the maximum likelihood estimator. The var-cov matrix of those estimates is an example of the Information identity at work.
A: I have to say that I am not sure what you are confused about.
"it seems like they would be zero or the same as the main diagonal?"
Not really - expected value of the score is zero but the cross products aren't (the different components of the score vector are not independent) - it is a matrix multiplication.
