# Why is the observed Fisher information defined as the Hessian of the log-likelihood?

In an MLE setting with probability density function $$f(X, \theta)$$, the (expected) Fisher information is usually defined as the covariance matrix of the fisher score, i.e. $$I(\theta) = E_\theta \left( \frac{\partial \log f(X; \theta)}{\partial \theta} \frac{\partial \log f(X; \theta)}{\partial \theta^T}\right).$$ Under the right regularity conditions, this is equivalent to $$I(\theta) = -E_{\theta}\left(\frac{\partial^2 \log f(X; \theta)}{\partial \theta^2} \right).$$

However, the observed Fisher information is always given as $$J(\theta) = -\frac{\partial^2 \log f(x; \theta)}{\partial \theta^2}.$$

Why is this the case? Why not consider $$\tilde{J}(\theta) = \frac{\partial \log f(x; \theta)}{\partial \theta} \frac{\partial \log f(x; \theta)}{\partial \theta^T}.$$

This answer and this one say the observed Fisher information is a consistent estimator of the expected Fisher information.

This leads me to the question summarized in the title, specifically:

• Why is the observed information always defined as the Hessian (analogous to the second definition of expected Fisher information above) and not using the gradient (as in the first definition)?
• Is $$\tilde{J}$$ also a consistent estimator of $$I$$?
• Why and in what sense is $$J$$ 'better' than $$\tilde{J}$$ when using it in practice? E.g. as basis for constructing confidence intervals.

Edit: I've discovered that $$\tilde{J}$$ is sometimes called the empirical Fisher information (McLachlan and Krishnan, 1997, Section 4.3). Still, I haven't found reasoning as to why this is inferior to $$J$$.

I find the literature in MLE a bit fuzzy with nomenclature here, so I might have some stuff off, and I will try to stick to the nomenclature you introduced.

We have the observed Fisher information:

$$\left[\mathcal {J}(\theta)\right]_{ij} = -\left(\frac{\partial^2 \log f}{\partial \theta_i \partial \theta_j}\right)$$

And the empirical Fisher information:

$$\left[\mathcal {\tilde J}(\theta)\right]_{ij} = \left(\frac{\partial \log f}{\partial \theta_i}\right)\left(\frac{\partial \log f}{\partial \theta_j}\right)$$

And it can be shown that with regularity (basically differentiability) conditions (see https://stats.stackexchange.com/a/101530/60613):

$$\left[\mathcal I(\theta)\right]_{ij} = E\left[\left[\mathcal J(\theta)\right]_{ij}\right] = E\left[\left[\mathcal {\tilde J}(\theta)\right]_{ij}\right]$$

So, why not use $$\mathcal {\tilde J}$$ instead of $$\mathcal J$$? Well, we actually use both.

The distinction is in that, using $$\mathcal {\tilde J}$$ (expected Hessian) for MLE we are doing IWLS (Fisher scoring), while $$\mathcal {J}$$ (observed Hessian) results in Newton-Raphson. $$\tilde {\mathcal J}$$ is guaranteed positive definite for non-overparametrized loglikelihoods (since you have more data than parameters, the covariance is full rank, see Why is the Fisher Information matrix positive semidefinite?), and the procedure benefits from that. $${\mathcal J}$$ does not enjoy of such benefits.

If we are performing MLE on the canonical parameter of a distribution in the exponential family, then both are actually identical.