In an MLE setting with probability density function $f(X, \theta)$, the (expected) Fisher information is usually defined as the covariance matrix of the fisher score, i.e. $$ I(\theta) = E_\theta \left( \frac{\partial \log f(X; \theta)}{\partial \theta} \frac{\partial \log f(X; \theta)}{\partial \theta^T}\right). $$ Under the right regularity conditions, this is equivalent to $$ I(\theta) = -E_{\theta}\left(\frac{\partial^2 \log f(X; \theta)}{\partial \theta^2} \right). $$

However, the observed Fisher information is always given as $$ J(\theta) = -\frac{\partial^2 \log f(x; \theta)}{\partial \theta^2}. $$

Why is this the case? Why not consider $$ \tilde{J}(\theta) = \frac{\partial \log f(x; \theta)}{\partial \theta} \frac{\partial \log f(x; \theta)}{\partial \theta^T}. $$

This answer and this one say the observed Fisher information is a consistent estimator of the expected Fisher information.

This leads me to the question summarized in the title, specifically:

  • Why is the observed information always defined as the Hessian (analogous to the second definition of expected Fisher information above) and not using the gradient (as in the first definition)?
  • Is $\tilde{J}$ also a consistent estimator of $I$?
  • Why and in what sense is $J$ 'better' than $\tilde{J}$ when using it in practice? E.g. as basis for constructing confidence intervals.

Edit: I've discovered that $\tilde{J}$ is sometimes called the empirical Fisher information (McLachlan and Krishnan, 1997, Section 4.3). Still, I haven't found reasoning as to why this is inferior to $J$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.