8
$\begingroup$

Wikipedia tells us that the score plays an important role in the Cramér–Rao inequality. It also phrases out the definition:

$$V = \frac{\partial}{\partial \theta} \log{L(\theta; X)}$$

However, I cannot find an intuitive explanation of what this quantity expresses. Obviously, it somehow measures how a small change of $\theta$ will affect the log-likelihood of the observed data $X$, but what exactly does that mean?

The wikipedia article also mentions that the expected value $\mathbb{E} [V \mid \theta] = 0$. Can this be interpreted somehow?

Going a bit further, in class we were told that the Fisher information (for which I have no intuitive understanding either) is $I(\theta) = \mathbb{E} [V^2 \mid \theta]$. Combined with $\mathbb{E} [V \mid \theta] = 0$ that would imply $I(\theta) = \text{Var}[V]$, is this correct?

Thanks in advance.

PS: This is not homework.

$\endgroup$
  • $\begingroup$ In regards to your last point, yes, Fisher information is the variance of the score. Can't help with intuition, though. $\endgroup$ – onestop Nov 24 '11 at 14:09
  • 2
    $\begingroup$ As far as intuition of the score, do you understand the intuition of a derivative? $\endgroup$ – user5594 Nov 24 '11 at 17:42
  • $\begingroup$ Strictly your final line should have been $I(\theta) = \text{Var}[V|\theta]$ $\endgroup$ – Henry Nov 25 '11 at 7:35
  • $\begingroup$ @MikeWierzbicki: Yes, of course. But I was hoping there was more to it... $\endgroup$ – blubb Nov 25 '11 at 13:33
2
$\begingroup$

The Wikipedia article gives an example of a Bernoulli process, with $A$ successes and $B$ failures and the probability of success $\theta$, where the score is $V = \frac{A}{\theta}-\frac{B}{1-\theta}$. If $\theta= \frac{A}{A+B}$, i.e. $\frac{\theta}{1-\theta}= \frac{A}{B}$, then $V=0$.

The score is more positive when there are more successes than would have been expected from the value of $\theta$, and more negative when there are fewer successes.

The score might be seen intuitively as a sort of measure of how close the parameter actually is to what the data suggest it might be (or the other way round if you are that way inclined), signed for the direction of the difference. The variance of the score will tend to increase with more data, so the variance is intuitively an indication of the amount of information the data will give you about the parameter.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.