In an MLE setting with probability density function $f(X, \theta)$, the (expected) Fisher information is usually defined as the covariance matrix of the fisher score, i.e. $$ I(\theta) = E_\theta \left( \frac{\partial \log f(X; \theta)}{\partial \theta} \frac{\partial \log f(X; \theta)}{\partial \theta^T}\right). $$ Under the right regularity conditions, this is equivalent to $$ I(\theta) = -E_{\theta}\left(\frac{\partial^2 \log f(X; \theta)}{\partial \theta^2} \right). $$
However, the observed Fisher information is always given as $$ J(\theta) = -\frac{\partial^2 \log f(x; \theta)}{\partial \theta^2}. $$
Why is this the case? Why not consider $$ \tilde{J}(\theta) = \frac{\partial \log f(x; \theta)}{\partial \theta} \frac{\partial \log f(x; \theta)}{\partial \theta^T}. $$
This answer and this one say the observed Fisher information is a consistent estimator of the expected Fisher information.
This leads me to the question summarized in the title, specifically:
- Why is the observed information always defined as the Hessian (analogous to the second definition of expected Fisher information above) and not using the gradient (as in the first definition)?
- Is $\tilde{J}$ also a consistent estimator of $I$?
- Why and in what sense is $J$ 'better' than $\tilde{J}$ when using it in practice? E.g. as basis for constructing confidence intervals.
Edit: I've discovered that $\tilde{J}$ is sometimes called the empirical Fisher information (McLachlan and Krishnan, 1997, Section 4.3). Still, I haven't found reasoning as to why this is inferior to $J$.
