Intuitive explanation of Fisher Information and Cramer-Rao bound

Question

I am not comfortable with Fisher information, what it measures and how is it helpful. Also it's relationship with the Cramer-Rao bound is not apparent to me.

Can someone please give an intuitive explanation of these concepts?

Answer 1

Here I explain why the asymptotic variance of the maximum likelihood estimator is the Cramer-Rao lower bound. Hopefully this will provide some insight as to the relevance of the Fisher information.

Statistical inference proceeds with the use of a likelihood function $\mathcal{L}(\theta)$ which you construct from the data. The point estimate $\hat{\theta}$ is the value which maximizes $\mathcal{L}(\theta)$. The estimator $\hat{\theta}$ is a random variable, but it helps to realize that the likelihood function $\mathcal{L}(\theta)$ is a "random curve".

Here we assume iid data drawn from a distribution $f(x|\theta)$, and we define the likelihood $$ \mathcal{L}(\theta) = \frac{1}{n}\sum_{i=1}^n \log f(x_i|\theta) $$

The parameter $\theta$ has the property that it maximizes the value of the "true" likelihood, $\mathbb{E}\mathcal{L}(\theta)$. However, the "observed" likelihood function $\mathcal{L}(\theta)$ which is constructed from the data is slightly "off" from the true likelihood. Yet as you can imagine, as the sample size increases, the "observed" likelihood converges to the shape of the true likelihood curve. The same applies to the derivative of the likelihood with respect to the parameter, the score function $\partial \mathcal{L}/\partial \theta$. (Long story short, the Fisher information determines how quickly the observed score function converges to the shape of the true score function.)

At a large sample size, we assume that our maximum likelihood estimate $\hat{\theta}$ is very close to $\theta$. We zoom into a small neighborhood around $\theta$ and $\hat{\theta}$ so that the likelihood function is "locally quadratic".

There, $\hat{\theta}$ is the point at which the score function $\partial \mathcal{L}/\partial \theta$ intersects the origin. In this small region, we treat the score function as a line, one with slope $a$ and random intercept $b$ at $\theta$. We know from the equation for a line that

$$a(\hat{\theta} - \theta) + b = 0$$

or

$$ \hat{\theta} = \theta - b/a . $$

From the consistency of the MLE estimator, we know that

$$ \mathbb{E}(\hat{\theta}) = \theta $$

in the limit.

Therefore, asymptotically

$$ nVar(\hat{\theta}) = nVar(b/a) $$

It turns out that the slope varies much less than the intercept, and asymptotically, we can treat the score function as having a constant slope in a small neighborhood around $\theta$. Thus we can write

$$ nVar(\hat{\theta}) = \frac{1}{a^2}nVar(b) $$

So, what are the values of $a$ and $nVar(b)$? It turns out that due to a marvelous mathematical coincidence, they are the very same quantity (modulo a minus sign), the Fisher information.

$$-a = \mathbb{E}\left[-\frac{\partial^2 \mathcal{L}}{\partial \theta^2}\right] = I(\theta)$$

$$nVar(b) = nVar\left[\frac{\partial \mathcal{L}}{\partial \theta}\right] = I(\theta)$$

Thus,

$$ nVar(\hat{\theta}) = \frac{1}{a^2}nVar(b) = (1/I(\theta)^2)I(\theta) = 1/I(\theta) $$ asymptotically: the Cramer-Rao lower bound. (Showing that $1/I(\theta)$ is a the lower bound on the variance of an unbiased estimator is another matter.)

Answer 2

One way that I understand the fisher information is by the following definition:

$$I(\theta)=\int_{\cal{X}} \frac{\partial^{2}f(x|\theta)}{\partial \theta^{2}}dx-\int_{\cal{X}} f(x|\theta)\frac{\partial^{2}}{\partial \theta^{2}}\log[f(x|\theta)]dx$$

The Fisher Information can be written this way whenever the density $f(x|\theta)$ is twice differentiable. If the sample space $\cal{X}$ does not depend on the parameter $\theta$, then we can use the Leibniz integral formula to show that the first term is zero (differentiate both sides of $\int_{\cal{X}} f(x|\theta)dx=1$ twice and you get zero), and the second term is the "standard" definition. I will take the case when the first term is zero. The cases when it isn't zero aren't much use for understanding Fisher Information.

Now when you do maximum likelihood estimation (insert "regularity conditions" here) you set

$$\frac{\partial}{\partial \theta}\log[f(x|\theta)]=0$$

And solve for $\theta$. So the second derivative says how quickly the gradient is changing, and in a sense "how far" $\theta$ can depart from the MLE without making an appreciable change in the right hand side of the above equation. Another way you can think of it is to imagine a "mountain" drawn on the paper - this is the log-likelihood function. Solving the MLE equation above tells you where the peak of this mountain is located as a function of the random variable $x$. The second derivative tells you how steep the mountain is - which in a sense tells you how easy it is to find the peak of the mountain. Fisher information comes from taking the expected steepness of the peak, and so it has a bit of a "pre-data" interpretation.

One thing that I still find curious is that its how steep the log-likelihood is and not how steep some other monotonic function of the likelihood is (perhaps related to "proper" scoring functions in decision theory? or maybe to the consistency axioms of entropy?).

The Fisher information also "shows up" in many asymptotic analysis due to what is known as the Laplace approximation. This basically due to the fact that any function with a "well-rounded" single maximum raise to a higher and higher power goes into a Gaussian function $\exp(-ax^{2})$ (similar to Central Limit Theorem, but slightly more general). So when you have a large sample you are effectively in this position and you can write:

$$f(data|\theta)=\exp(\log[f(data|\theta)])$$

And when you taylor expand the log-likelihood about the MLE:

$$f(data|\theta)\approx [f(data|\theta)]_{\theta=\theta_{MLE}}\exp\left(-\frac{1}{2}\left[-\frac{\partial^{2}}{\partial \theta^{2}}\log[f(data|\theta)]\right]_{\theta=\theta_{MLE}}(\theta-\theta_{MLE})^{2}\right)$$ and that second derivative of the log-likelihood shows up (but in "observed" instead of "expected" form). What is usually done here is to make to further approximation:

$$-\frac{\partial^{2}}{\partial \theta^{2}}\log[f(data|\theta)]=n\left(-\frac{1}{n}\sum_{i=1}^{n}\frac{\partial^{2}}{\partial \theta^{2}}\log[f(x_{i}|\theta)]\right)\approx nI(\theta)$$

Which amounts to the usually good approximation of replacing a sum by an integral, but this requires that the data be independent. So for large independent samples (given $\theta$) you can see that the Fisher information is how variable the MLE is, for various values of the MLE.

Answer 3

Although the explanations provided above are very interesting and I've enjoyed going through them, I feel that the nature of the Cramer-Rao Lower Bound was best explained to me from a geometric perspective. This intuition is a summary of the concept of concentration ellipses from Chapter 6 of Scharf's book on Statistical Signal Processing.

Consider any unbiased estimator of ${\boldsymbol\theta}$. Additionally, assume that the estimator $\hat{\boldsymbol\theta}$ has a Gaussian distribution with covariance ${\boldsymbol\Sigma}$. Under these conditions, the distribution of $\hat{\boldsymbol\theta}$ is proportional to:

$f(\hat{\boldsymbol\theta})\propto \exp(-\frac{1}{2}(\hat{\boldsymbol\theta}-{\boldsymbol\theta})^T{\boldsymbol\Sigma}^{-1}(\hat{\boldsymbol\theta}-{\boldsymbol\theta}))$.

Now think of the contour plots of this distribution for ${\boldsymbol\theta}\in R^2$. Any upper bound constraint on the probability of $\hat{\boldsymbol\theta}$ (i.e., $\int f(\hat{\boldsymbol\theta})d{\boldsymbol\theta} \le P_r$) will result in an ellipsoid centered at ${\boldsymbol\theta}$ with fixed radius $r$. It's easy to show that there is a one-to-one relationship between the radius $r$ of the ellipsoid and the desired probability $P_r$. In other words, $\hat{\boldsymbol\theta}$ is close to ${\boldsymbol\theta}$ within an ellipsoid determined by radius $r$ with probability $P_r$. This ellipsoid is called a concentration ellipsoid.

Considering the description above, we can say the following about the CRLB. Among all unbiased estimators, the CRLB represents an estimator $\hat{\boldsymbol\theta}_{crlb}$ with covariance $\boldsymbol\Sigma_{crlb}$ that, for fixed probability of "closeness" $P_r$ (as defined above), has the smallest concentration ellipsoid. The Figure below provides a 2D illustration (inspired by illustration in Scharf's book).

Answer 4

This is the most intuitive article that I have seen so far:

The Cramér-Rao Lower Bound on Variance: Adam and Eve’s “Uncertainty Principle” by Michael R. Powers, Journal of Risk Finance, Vol. 7, No. 3, 2006

The bound is explained by an analogy of Adam and Eve in the Garden of Eden tossing a coin to see who gets to eat the fruit and they then ask themselves just how big a sample is necessary to achieve a certain level of accuracy in their estimate, and they then discover this bound...

Nice story with a profound message about reality indeed.