Fisher's score function has mean zero - what does that even mean?

Question

I'm trying to follow the princeton review of likelihood theory. They define Fisher’s score function as The first derivative of the log-likelihood function, and they say that the score is a random vector. E.g for the Geometric distribution:

$$ u(\pi) = n\left(\frac{1}{\pi} - \frac{\bar{y}}{1-\pi} \right) $$

And I can see that it is indeed a function (of the parameter $\pi$), and it is random, as it involves $\bar{y}$.

BUT then they say something I don't understand: "the score evaluated at the true parameter value $\pi$ has mean zero" and they formulate it as $E(u(\pi)) = 0$. What does it mean to evaluate it at the "true parameter value" and then find out its mean? And in the Geometric example, if I use the identity $E(y) = E(\bar{y}) = \frac{1-\pi}{\pi}$ won't I immediately get that $E(u(\pi)) = 0$? what does the "true parameter value" has to do with this?

Answer 1

As you pointed out the score function $u$ is, under suitable regularity conditions, defined as "the first derivative of the log-likelihood function".

Let's suppose that $X$ is a random variable with density function $f(x)$. Usually this density changes depending on a vector of parameters $\pi$. Thus it is convenient to wright the density function as $f(x;\pi)$ to make it explicit the dependency on the parameter. We will suppose that the "true" value of $\pi$ for the random variable $X$ is $\pi = \pi_0$. (what I mean is that $X \sim f(\;\cdot\;;\pi_0)$)

The score function can now be written as: $$u(\pi;x) = \frac{\partial}{\partial\pi}\log f(x;\pi),$$ and it is now clear that it is a function of both $x$ and of $\pi$. (In your question you have $L$ in place of $f$, but there is no difference since the likelihood function is just the density function.)

Consider now the random variable $u(\pi,X)$ and its expectation $\xi(\pi) = \mathbb{E_{\pi_0}}(u(\pi,X))$. Here it is important to notice that the subscript $\pi_0$ is there to indicate the (true) parameter in the distribution of $X$ and diferentiate it from the value $\pi$ with which we are calculating $u$.

Assuming that $f$ is a continuous density (the discrete case is similar) we have:

$$\xi(\pi) = \int_{-\infty}^{+\infty}\left(\frac{\partial}{\partial\pi}\log f(x;\pi)\right)f(x;\pi_0)dx = \int_{-\infty}^{+\infty}\frac{f'(x;\pi)}{f(x;\pi)}f(x;\pi_0)dx$$

and when you evaluate $\xi$ at the true parameter value $\pi_0$ we get:

$$\xi(\pi_0) = \int_{-\infty}^{+\infty}\frac{f'(x;\pi_0)}{f(x;\pi_0)}f(x;\pi_0)dx = \int_{-\infty}^{+\infty}f'(x;\pi_0)dx$$ $$=\frac{\partial}{\partial\pi}\int_{-\infty}^{+\infty}f(x;\pi_0)dx = 0$$

That's the reasoning behind the score function having expectation zero at the true parameter.

You should take a look at books like this one (chapter 3) to have a bigger understanding of the conditions under which those derivations (like the interchanging of derivative and integral) hold true.

Answer 2

Ok, thanks to the excellent @Mur1lo answer I now have a better understanding and would like to make my own attempt at making this abstract concept as concrete as I can.

Suppose we have a sample of 5 coin draw results. We assume that they are sampled from a population with Bernoulli distribution with the true parameter $\pi_0$.

When we look at a specific coin draw with result $x_3=1$, we can calculate the log-likelihood this patient was sampled from a Bernoulli distribution with all kind of parameter values, e.g. $\pi = 0.2$ or $\pi=0.9$ and so on. so, log-likelihood is a function estimating the likelihood of $x_3$ for each possible value of $\pi$.

$$ LL(\pi|x_3) = x_3ln(\pi) + (1-x_3)ln(1-\pi) $$

Which simply means that if $x_3=1$ the likelihood for that was $\pi$ and if it's 0 the likelihood for that is $1-\pi$.

If we assume independence between the coin draws, then we have an 'average' function representing the log-likelihood of the entire sample of n=5 coin draws.

$$ LL(\pi|X) = \sum{x_i}ln(\pi) + (n-\sum(x_i))ln(1-\pi) $$

We want to find the maximum of $LL(\pi|X)$ - the mle = $\pi_{mle}$.

The score function $u(\pi)$ is a vector of the derivatives w.r.t each parameter of the log-likelihood. Luckily in our case, it's a simple scalar as there's only one parameter. Under some conditions, it will help us find $\pi_{mle}$, since in that point the score function would be $u(\pi_{mle}) = 0$. We can calculate the observation score function for a single observation (coin draw):

$$ u(\pi|x_3) = \frac{x_3}{\pi} - \frac{1-x_3}{1-\pi} $$

and the sample score function of n=5 patients:

$$ u(\pi|X) = \frac{\sum{x_i}}{\pi} - \frac{n-\sum{x_i}}{1-\pi} $$

when we set this latest function to 0, we get $\pi_{mle}$.

---

BUT, the specific 5 draws sample has nothing to do with the expectancy of the score function ! The expectancy is the value of the observation score function for every possible value of x, multiplied by the probability of that value, which is the density function! In our case, x can take only 2 values: 0 and 1. And the density function is as we assumed is a Bernoulli with parameter $\pi_0$:

$$ E(u(\pi|x_i)) = \sum_x (\frac{x}{\pi} - \frac{1-x}{1-\pi}) \pi_0^x(1-\pi_0)^{1-x} = \frac{\pi_0}{\pi} - \frac{1-\pi_0}{1-\pi}$$

and its clear that it zeros out when evaluated at the true parameter $\pi_0$. The intuitive interpretation is: For each value of $\pi$, what's the mean rate of change in likelihood?

---

The information matrix is the variance of the likelihood - how sensitive will our solution be to different data? (see this answer).

$$I(\pi|x_i) = var(u(\pi|x_i)) = var(\frac{x_i}{\pi} - \frac{1-x_i}{1-\pi}) = var(\frac{x_i-\pi}{\pi(1-\pi)}) = \frac{var(x_i)}{\pi^2(1-\pi)^2} = \frac{\pi_0(1-\pi_0)}{\pi^2(1-\pi)^2}$$

and when evaluated at the true parameter $\pi_0$ it simplifies to:

$$I(\pi_0|x_i) = \frac{1}{\pi_0(1-\pi_0)}$$

(see washington edu notes for more details).

Amazingly, there's another way of measuring how sensitive the likelihood would be in a certain $\pi$! that's the expectancy of the curvature = Hessian = second derivative. The steeper our likelihood, the more accurate we'll be. See details in mark reid's blog