5
$\begingroup$

I'm following Kashin's review of MLE

Suppose we have a simple Bernoulli distribution and we know its true parameter $p_0$. The Fisher expected information $I(p_0)$ is equal to the variance of the expected score, when evaluated at $p_0$.

It's also known that the MLE estimate $p_{mle}$ converges to $p_0$ like $N(p_0, I(p_0)^{-1})$. So as LARGER the information gets, the variance of the score gets LARGER, but our confidence in the MLE gets better! This doesn't make sense to me intuitively:

In the below graphs the X-axis is values of $p$, and both the log-likelihood $L$ and the score $S$ are plotted for 2 samples. On the left graph they have a larger observed fisher information than on the right, so their curvatures are steeper and indeed the score when evaluated at $p_0$ has a larger variance (in green).

So why is it that we are more confident that the MLEs on the left graph will converge towards $p_0$? We can see that their score variance in $p_0$ is larger, which means that they are farther away from the expected score $E(S(p_0)) = 0$.

Kashin has a formal proof, can anyone lay an intuitive explanation on this counter-intuitive (at least for me) phenomenon?

NOTE: I realize that the curvature of the likelhoods on the left is steeper than those on the right, and I understand the intuition behind this interpretation of the information - It is the other interpretation of the information - the score variance when evaluated at the true parameter - which I want to understand.

Thanks!

mle

$\endgroup$
2
$\begingroup$

I'm not sure what is really puzzling you. The score indicates how sensitively a likelihood function depends on its parameter. So it grows (and so does the variance) with the amount of information that the likelihood carries about the parameter. On the other, the maximum likelihood estimate get better as the amount of information grows. Seems intuitive enough for me...

I made some maths to clarify things. You can also have a look at my notebook where I checked that the maths are working by generating some Bernoulli samples.

Math

Let $X_1 ,...,X_T$ be an iid sample with $X_i \sim Bernoulli(\theta_0)$ with $\theta_0$ being the true parameter.

The joint density/likelihood function is given by: $$f( x ; \theta )= L ( \theta | x )= \prod_{i=1}^{T}{\theta^{x_i}(1-\theta)^{1-x_i}}=\theta^{\sum x_i}(1-\theta)^{n-\sum x_i}$$

The score function is given by: $$ S(\theta | x)=\frac{\partial \ln L(\theta|x)}{\partial \theta}= \frac{1}{\theta}\sum x_i - \frac{1}{1-\theta}(T - \sum x_i) $$

The MLE of $\theta_0$ is: $$\hat{\theta}=\frac{1}{T}\sum_{i=1}^{T}x_i $$

And the sample Fisher Information is: $$ I(\theta | x) = -\mathbf{E}[\frac{\partial^2 \ln L(\theta|x)}{\partial \theta^2}]$$

Evaluated at $\theta_0$: $$ I(\theta_0 | x) = T(\frac{1}{\theta_0} + \frac{1}{1-\theta_0}) $$

Then: $$\frac{1}{\sqrt{T}}S(\theta_0|x) \xrightarrow{D} \mathcal{N}(0, \frac{I(\theta_0)}{T}) $$

and: $$\sqrt{T}(\hat{\theta}-\theta_0) \xrightarrow{D} \mathcal{N}(0, \{\frac{I(\theta_0)}{T}\}^{-1}) $$

Conclusion

  • Situation 1.

$ I(\theta_0 | x) = -\mathbf{E}[\frac{\partial^2 \ln L(\theta|x)}{\partial \theta^2}\big|_{\theta_0}] $ is large (hence variance of the mle will be small) then it means that the gradient of $\frac{\partial \ln L(\theta|x)}{\partial \theta}$ is steep. Hence even for small deviations from $\theta_0$ ,$\frac{\partial \ln L(\theta|x)}{\partial \theta}$ is likely to be far from zero. This means the mle $\hat{\theta}$ is likely to be in a close neighbourhood of $\theta_0$.

  • Situation 2.

$ I(\theta_0 | x) = -\mathbf{E}[\frac{\partial^2 \ln L(\theta|x)}{\partial \theta^2}\big|_{\theta_0}] $ is small (hence variance of the mle will be large). In this case the gradient of the likelihood $\frac{\partial \ln L(\theta|x)}{\partial \theta}$ is flatter and hence $\frac{\partial \ln L(\theta|x)}{\partial \theta}\approx 0$ for a large neighbourhood about the true parameter $\theta_0$. Therefore the mle $\hat{\theta}$ can lie in a large neighbourhood of $\theta_0$.

Sources

http://faculty.washington.edu/ezivot/econ583/mleLectures.pdf

https://www.stat.tamu.edu/~suhasini/teaching613/STAT613.pdf

$\endgroup$
1
  • $\begingroup$ Wow, @Adrien - you went to a lot of trouble for me and I really appreciate it! However, I'm still not sure what am I seeing there - I was interested in the variance of the score - and it looks like its the same = 11 in both situations. Also the histograms spread is about the same... Also - I think that the core of what you're saying is in the end - can you explain this to me "Hence even for small deviations from θ0 ,the score is likely to be far from zero. This means that the mle is likely to be in a close neighbourhood of θ0." - thanks again!!! $\endgroup$
    – ihadanny
    Sep 12 '16 at 19:57
1
$\begingroup$

My intuition would be as follows: As $n\to\infty$, $p_{mle}\to_p p_0$ by consistency of the MLE, so the log-likelihood will be centered around the true value.

Now, if we have more curvature, that means that the log-likelihood bends away from the true value more quickly, meaning that other candidate values $p$ for the true value $p_0$ are assigned much less likelihood than the MLE, meaning that we are much more confident to prefer the MLE over other values of $p$.

Conversely, if the log-likelihood is relatively flat like on the right, we still have that, by construction, the MLE is at the peak of the log-likelihood, but other values receive about the same likelihood, so that we would not be very confident to prefer the MLE over the others.

$\endgroup$
2
  • $\begingroup$ Thanks! but your intuition relies solely on the curvature of the log-likelihood which is one interpretation for the fisher information, you ignored the score variance at the true parameter, which is another interpretation of the fisher information - and that's what my question is about... $\endgroup$
    – ihadanny
    Sep 9 '16 at 13:30
  • 1
    $\begingroup$ Fair point - here, my intuition to relate these two ideas would be that, if the curvature is higher, a small change in the data will imply a large chance in the score, because even if we move just a little along the axis, the slope will change a lot. $\endgroup$ Sep 9 '16 at 13:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.