This question already has an answer here:

(We may just assume single parameter model and single variable data for this discussion).

I understand that the "observed information" evaluated at the MLE measures the curvature and hence gives a gauge of how much information our data is giving us. "Observed information" here refers to having already obtained a sample and then computing $-d^2l/d\theta^2$ ($l$ here is log-likelihood).

I also understand the definition of Fisher Information, and I understand that $l$ is random before the data is drawn. However, in terms of an intuitive explanation, Wikipedia and various other sources say that Fisher Information gives a measure of curvature or still tells us information in the same way as I described for observed information. This, I disagree. When I evaluate $I(\theta)$ by average the curvature over all possible $l$, we must keep in mind that $\theta$ need not be the peak of $l$ - for some $l$, it is, but for other $l$, we may be way, way away from the peak, and the curvature that these scenerios contribute to the Fisher Information may lead to funny effects. Why would we care about the curvature at $\theta$ for these weird $l$ where $\theta$ is clearly not a good estimation?

Basically, I am not very convinced on the intuitive explanation given for Fisher Information. Is the intuitive explanation commonly given a correct one?


marked as duplicate by kjetil b halvorsen, mdewey, Peter Flom - Reinstate Monica Apr 19 '18 at 13:40

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ I think you should first state a formal definition before declaring that you disagree with it. Also, what you forgot to mention is that the curvature of the log likelihood is evaluated AT the maximum likelihood estimate. This is exact only in the infinite sample size limit, or when the sample process is truly Gaussian. $\endgroup$ – ocramz Nov 17 '15 at 8:28