58
$\begingroup$

I am not comfortable with Fisher information, what it measures and how is it helpful. Also it's relationship with the Cramer-Rao bound is not apparent to me.

Can someone please give an intuitive explanation of these concepts?

$\endgroup$
  • 1
    $\begingroup$ Is there anything in the Wikipedia article which is causing problems? It measures the amount of information that an observable random variable $X$ carries about an unknown parameter $\theta$ upon which the probability of $X$ depends, and its inverse is the Cramer-Rao lower bound on the variance of an unbiased estimator of $\theta$. $\endgroup$ – Henry May 9 '11 at 20:55
  • 2
    $\begingroup$ I understand that but I am not really comfortable with it. Like, what exactly does "amount of information" means here. Why does negative expectation of the square of the partial derivative of the density measures this information? Where does the expression come from etc. That's why I am hoping to get some intuition about it. $\endgroup$ – Infinity May 9 '11 at 22:18
  • $\begingroup$ @Infinity: The score is the proportional rate of change in the likelihood of the observed data as the parameter changes, and so useful for inference. The Fisher information the variance of the (zero-meaned) score. So mathematically it is the expectation of the square of the first partial derivative of the logarithm of the density and so is the negative of the expectation of the second partial derivative of the logarithm of the density. $\endgroup$ – Henry May 9 '11 at 23:07
32
$\begingroup$

Here I explain why the asymptotic variance of the maximum likelihood estimator is the Cramer-Rao lower bound. Hopefully this will provide some insight as to the relevance of the Fisher information.

Statistical inference proceeds with the use of a likelihood function $\mathcal{L}(\theta)$ which you construct from the data. The point estimate $\hat{\theta}$ is the value which maximizes $\mathcal{L}(\theta)$. The estimator $\hat{\theta}$ is a random variable, but it helps to realize that the likelihood function $\mathcal{L}(\theta)$ is a "random curve".

Here we assume iid data drawn from a distribution $f(x|\theta)$, and we define the likelihood $$ \mathcal{L}(\theta) = \frac{1}{n}\sum_{i=1}^n \log f(x_i|\theta) $$

The parameter $\theta$ has the property that it maximizes the value of the "true" likelihood, $\mathbb{E}\mathcal{L}(\theta)$. However, the "observed" likelihood function $\mathcal{L}(\theta)$ which is constructed from the data is slightly "off" from the true likelihood. Yet as you can imagine, as the sample size increases, the "observed" likelihood converges to the shape of the true likelihood curve. The same applies to the derivative of the likelihood with respect to the parameter, the score function $\partial \mathcal{L}/\partial \theta$. (Long story short, the Fisher information determines how quickly the observed score function converges to the shape of the true score function.)

At a large sample size, we assume that our maximum likelihood estimate $\hat{\theta}$ is very close to $\theta$. We zoom into a small neighborhood around $\theta$ and $\hat{\theta}$ so that the likelihood function is "locally quadratic".

There, $\hat{\theta}$ is the point at which the score function $\partial \mathcal{L}/\partial \theta$ intersects the origin. In this small region, we treat the score function as a line, one with slope $a$ and random intercept $b$ at $\theta$. We know from the equation for a line that

$$a(\hat{\theta} - \theta) + b = 0$$

or

$$ \hat{\theta} = \theta - b/a . $$

From the consistency of the MLE estimator, we know that

$$ \mathbb{E}(\hat{\theta}) = \theta $$

in the limit.

Therefore, asymptotically

$$ nVar(\hat{\theta}) = nVar(b/a) $$

It turns out that the slope varies much less than the intercept, and asymptotically, we can treat the score function as having a constant slope in a small neighborhood around $\theta$. Thus we can write

$$ nVar(\hat{\theta}) = \frac{1}{a^2}nVar(b) $$

So, what are the values of $a$ and $nVar(b)$? It turns out that due to a marvelous mathematical coincidence, they are the very same quantity (modulo a minus sign), the Fisher information.

$$-a = \mathbb{E}\left[-\frac{\partial^2 \mathcal{L}}{\partial \theta^2}\right] = I(\theta)$$

$$nVar(b) = nVar\left[\frac{\partial \mathcal{L}}{\partial \theta}\right] = I(\theta)$$

Thus,

$$ nVar(\hat{\theta}) = \frac{1}{a^2}nVar(b) = (1/I(\theta)^2)I(\theta) = 1/I(\theta) $$ asymptotically: the Cramer-Rao lower bound. (Showing that $1/I(\theta)$ is a the lower bound on the variance of an unbiased estimator is another matter.)

$\endgroup$
  • 2
    $\begingroup$ Is there any graphical representation of the part where you mention that the likelihood function is locally quadratic? $\endgroup$ – quirik May 30 '17 at 11:03
  • $\begingroup$ @quirik , consider using the second order Taylor expansion around theta_hat. $\endgroup$ – idnavid Jul 27 '18 at 3:00
  • $\begingroup$ @charles.y.zheng This is one of the most interesting explanations of scene. $\endgroup$ – idnavid Jul 27 '18 at 3:17
13
$\begingroup$

One way that I understand the fisher information is by the following definition:

$$I(\theta)=\int_{\cal{X}} \frac{\partial^{2}f(x|\theta)}{\partial \theta^{2}}dx-\int_{\cal{X}} f(x|\theta)\frac{\partial^{2}}{\partial \theta^{2}}\log[f(x|\theta)]dx$$

The Fisher Information can be written this way whenever the density $f(x|\theta)$ is twice differentiable. If the sample space $\cal{X}$ does not depend on the parameter $\theta$, then we can use the Leibniz integral formula to show that the first term is zero (differentiate both sides of $\int_{\cal{X}} f(x|\theta)dx=1$ twice and you get zero), and the second term is the "standard" definition. I will take the case when the first term is zero. The cases when it isn't zero aren't much use for understanding Fisher Information.

Now when you do maximum likelihood estimation (insert "regularity conditions" here) you set

$$\frac{\partial}{\partial \theta}\log[f(x|\theta)]=0$$

And solve for $\theta$. So the second derivative says how quickly the gradient is changing, and in a sense "how far" $\theta$ can depart from the MLE without making an appreciable change in the right hand side of the above equation. Another way you can think of it is to imagine a "mountain" drawn on the paper - this is the log-likelihood function. Solving the MLE equation above tells you where the peak of this mountain is located as a function of the random variable $x$. The second derivative tells you how steep the mountain is - which in a sense tells you how easy it is to find the peak of the mountain. Fisher information comes from taking the expected steepness of the peak, and so it has a bit of a "pre-data" interpretation.

One thing that I still find curious is that its how steep the log-likelihood is and not how steep some other monotonic function of the likelihood is (perhaps related to "proper" scoring functions in decision theory? or maybe to the consistency axioms of entropy?).

The Fisher information also "shows up" in many asymptotic analysis due to what is known as the Laplace approximation. This basically due to the fact that any function with a "well-rounded" single maximum raise to a higher and higher power goes into a Gaussian function $\exp(-ax^{2})$ (similar to Central Limit Theorem, but slightly more general). So when you have a large sample you are effectively in this position and you can write:

$$f(data|\theta)=\exp(\log[f(data|\theta)])$$

And when you taylor expand the log-likelihood about the MLE:

$$f(data|\theta)\approx [f(data|\theta)]_{\theta=\theta_{MLE}}\exp\left(-\frac{1}{2}\left[-\frac{\partial^{2}}{\partial \theta^{2}}\log[f(data|\theta)]\right]_{\theta=\theta_{MLE}}(\theta-\theta_{MLE})^{2}\right)$$ and that second derivative of the log-likelihood shows up (but in "observed" instead of "expected" form). What is usually done here is to make to further approximation:

$$-\frac{\partial^{2}}{\partial \theta^{2}}\log[f(data|\theta)]=n\left(-\frac{1}{n}\sum_{i=1}^{n}\frac{\partial^{2}}{\partial \theta^{2}}\log[f(x_{i}|\theta)]\right)\approx nI(\theta)$$

Which amounts to the usually good approximation of replacing a sum by an integral, but this requires that the data be independent. So for large independent samples (given $\theta$) you can see that the Fisher information is how variable the MLE is, for various values of the MLE.

$\endgroup$
  • 1
    $\begingroup$ "One thing that I still find curious is that its how steep the log-likelihood is and not how steep some other monotonic function of the likelihood is." I'm sure you could derive analogues for Fisher information in terms of other transformations of the likelihood, but then you wouldn't get as neat of an expression for the Cramer-Rao lower bound. $\endgroup$ – charles.y.zheng May 10 '11 at 15:11
2
$\begingroup$

This is the most intuitive article that I have seen so far:

The Cramér-Rao Lower Bound on Variance: Adam and Eve’s “Uncertainty Principle” by Michael R. Powers, Journal of Risk Finance, Vol. 7, No. 3, 2006

The bound is explained by an analogy of Adam and Eve in the Garden of Eden tossing a coin to see who gets to eat the fruit and they then ask themselves just how big a sample is necessary to achieve a certain level of accuracy in their estimate, and they then discover this bound...

Nice story with a profound message about reality indeed.

$\endgroup$
  • 6
    $\begingroup$ Thank you for posting this reference. At the end I was disappointed, though, to find that it doesn't actually explain the CRLB. It merely states it, without providing any insight into why it's true, and only provide some evocative but ultimately meaningless language, like "squeezing information," in an effort to explain it. $\endgroup$ – whuber Aug 3 '16 at 16:40
  • $\begingroup$ @whuber: Fair enough, I agree that it could dive in deeper and the ending is a little abrupt. Yet what I like about the article is that it really seems natural that there is a connection between sample size, sample mean, the law of large numbers and that the sample variance can only be reduced up to a point (i.e. that there has to be some bound, which happens to be the abovementioned one). It also makes clear that this is not some elusive mathematical result but really a statement about the limits of gaining knowledge of reality. $\endgroup$ – vonjd Aug 3 '16 at 17:25
2
$\begingroup$

Although the explanations provided above are very interesting and I've enjoyed going through them, I feel that the nature of the Cramer-Rao Lower Bound was best explained to me from a geometric perspective. This intuition is a summary of the concept of concentration ellipses from Chapter 6 of Scharf's book on Statistical Signal Processing.

Consider any unbiased estimator of ${\boldsymbol\theta}$. Additionally, assume that the estimator $\hat{\boldsymbol\theta}$ has a Gaussian distribution with covariance ${\boldsymbol\Sigma}$. Under these conditions, the distribution of $\hat{\boldsymbol\theta}$ is proportional to:

$f(\hat{\boldsymbol\theta})\propto \exp(-\frac{1}{2}(\hat{\boldsymbol\theta}-{\boldsymbol\theta})^T{\boldsymbol\Sigma}^{-1}(\hat{\boldsymbol\theta}-{\boldsymbol\theta}))$.

Now think of the contour plots of this distribution for ${\boldsymbol\theta}\in R^2$. Any upper bound constraint on the probability of $\hat{\boldsymbol\theta}$ (i.e., $\int f(\hat{\boldsymbol\theta})d{\boldsymbol\theta} \le P_r$) will result in an ellipsoid centered at ${\boldsymbol\theta}$ with fixed radius $r$. It's easy to show that there is a one-to-one relationship between the radius $r$ of the ellipsoid and the desired probability $P_r$. In other words, $\hat{\boldsymbol\theta}$ is close to ${\boldsymbol\theta}$ within an ellipsoid determined by radius $r$ with probability $P_r$. This ellipsoid is called a concentration ellipsoid.

Considering the description above, we can say the following about the CRLB. Among all unbiased estimators, the CRLB represents an estimator $\hat{\boldsymbol\theta}_{crlb}$ with covariance $\boldsymbol\Sigma_{crlb}$ that, for fixed probability of "closeness" $P_r$ (as defined above), has the smallest concentration ellipsoid. The Figure below provides a 2D illustration (inspired by illustration in Scharf's book).

2D illustration of CRLB in the context of unbiased estimators.

$\endgroup$
  • 2
    $\begingroup$ Well this is bloody great, especially the image, needs more upvotes. $\endgroup$ – Astrid Nov 27 '18 at 22:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.