Why is the Cramer-Rao Lower Bound (CRLB) inverse of the Fisher Information I(θ)?

Question

Why is the Cramer-Rao Lower Bound (CRLB) inverse of the Fisher Information I(θ)? Could someone provide an intuitive explanation? I am having trouble understanding the concept.

Answer 1

Almost two years later comes the longer answer: This is not a rigorous explanation but hopefully gives some intuition that the variance of the ML-estimator increases with the curvature of the log-likelihood (at least in the following simple example).

Assume that we have $m$ samples of size $n$ from $N(0, \sigma_1^2)$ and from $N(0, \sigma_2^2)$ where I pick $\sigma_1^2 = 1$ and $\sigma_2^2 = 2$.

The following graphs depict the log-likelihoods for these samples for $n = 20$ and $m=10$ (assuming we know the variance). The left side shows the samples for $\sigma_1^2$ and the right graph for $\sigma_2^2$

Now the ML-estimator for one of these samples is the argmax of the respective function.

We can observe that:

1. The ML-estimators on the left side have less variance than the ML-estimators on the right side
2. The curvature of the graphs on the left side is considerable higher than the curvature on the right side.

So it seems to be that the curvature of the log-likelihood is inversely proportional to the variability of the ML-estimator.

So how can we make sense of this on an intuitive level?

If we consider the unimodal shape of the log-likelihood (which eventuall happens for most distributions once we have enough samples) the curvature of the log-likelihood gives an indication of how far away we can move from the best explanation of our current sample (i.e. the ML-estimate) and still get an almost as good as explanation (meaning a log-likelihood not much different). When we can move far away from our ML-estimate and still get a similarly good explanation, we should not be surprised to find out that in our next sample the best explanation is a very different one as in our current sample. If the curvature is very high and changes in paramters lead to a greater degree of explaindness (meaning log-likelihood) we would expect the best-guesses in other samples to be rather close to the best explanation for the current situation.

Answer 2

I think this YouTube video by Ben Lambert gives a neat intuition, as it discusses the Cramer Rao Bound and Fisher information in a simple case in which geometric intuitions still work.

Answer 3

There is a certain correspondence between the variance of the estimator and the variance of the score or derivative of the likelihood. This becomes possibly more clear when we slightly rewrite the expression for the Cramer Rao bound instead of $\text{var}( \hat{\theta} ) \geq \frac{1}{I(\theta)}$ we can write it also as

$$\text{var}\left( \hat{\theta} \right) \cdot I(\theta) = \text{var}\left( \hat{\theta} \right) \cdot \text{var}\left( \frac{f^\prime({\bf x}; \theta)}{f({\bf x}; \theta)} \right) \geq 1 $$

where we use the notation with a prime to denote a derivative with $\theta$, ie. $f^\prime(x,\theta) = \partial f(x,\theta)/\partial \theta$

from this expression it might become clear wherefrom this inverse gets into the equation. We can derive it as an upper limit for the product of the estimator variance and the score variance.

The background of the above inequality is more precisely related to how the change in the expectation value of the estimator relates to the variance of the estimator and the Fisher information matrix

$$\text{var}\left( \hat{\theta}({\bf x}) \right) \cdot \text{var}\left( \frac{f^\prime({\bf x}; \theta)}{f({\bf x}; \theta)} \right) \geq \left(\text{cov}\left(\hat\theta({\bf x}) , \frac{f^\prime({\bf x}; \theta)}{f({\bf x}; \theta)} \right)\right)^2 = \left(\frac{\partial E[\hat{\theta}]}{\partial\theta}\right)^2\underbrace{ = 1 \vphantom{\frac{\partial E[\hat{\theta}]}{\partial\theta}}}_{\substack{\llap{\text{for unbiased estima}} \rlap{\text{tors we have $E[\hat{\theta}] = \theta$}} \\ \llap{\text{and the deriv}} \rlap{\text{ative equals 1}}} }$$

This interaction between the $\hat{\theta}({\bf x})$ and the score $\frac{f^\prime({\bf x}; \theta)}{f({\bf x}; \theta)}$ can be seen as analogous to a torque in physics. The change of the expectation value of the estimator $\frac{\partial E[\hat{\theta}]}{\partial\theta}$ relates to the change of the density function $\frac{f^\prime({\bf x}; \theta)}{f({\bf x}; \theta)}$ (the force in the torque analogy) and the place $ \hat{\theta}$ where this change occurs (the distance/arm in the torque analogy).

The image below, with three layers, tries to illustrate this. The image is one for two observed values $x_1,x_2$

• In the top layer it shows two probability density distribution for two different $\theta_1$ and $\theta_2$.
• In the middle layer we see the change/difference of these two distributions.
• In the bottom layer we see the estimated value The expectation value $\hat{\theta}$ as function of the observation $x_1,x_2$.

For the two distributions the expectation value of $\hat{\theta}$ is different. How much this difference in expectation value is, depends on the rate of change of the density functions, and the at which values $\hat{\theta}$ this happens. If the density functions change for large and small values of $\hat{\theta}$, then the change of $E[\hat{\theta}]$ will be stronger.

Since the change $E[\hat{\theta}]$ is constrained (we must have $\partial E[\hat{\theta}]/\partial \theta = 1$ otherwise the estimator is not unbiased) we have that for a distribution with a large Fisher information matrix (change of information at large distance), the sample distribution of the estimate $\hat{\theta}$ can not be too much spread out.