# Derivative of the Score Function in Fisher Information

I'm studying Fisher Information and am trying to develop an intuitive understanding. Keep in mind I only have bachelor level mathematics background so I would appreciate an answer that is more intuitively explained rather than mathematically derived.

I understand we use Fisher Information in to give us some sense of "confidence" around our MLE. However, I'm confused mathematically how we could possibly get any fisher information other than zero at the MLE. Since FI is equal to expected value (wrt X) of the square of the first derivative of the score function, the derivative at a theta equal to the MLE is simply 0. And 0 squared is equal to 0. How is it possible to ever have FI that is meaningful at the MLE then? Does it have to do with the fact that we are averaging the derivative across all values of our random variable X, and therefore the slope may be non-zero in some cases?

Any insight is appreciated.

This is an interesting confusion --- it comes from the fact that the MLE for a fixed set of data has a zero score (under standard regularity conditions) but data randomly generated from a simulation model using the MLE as the parameter will not generally have a zero score at that parameter value. In other words, the MLE from a fixed set of data is not generally going to also be the MLE from a randomly generated set of data that is generated from a model using that MLE as the parameter value.

In relation to your present confusion about the Fisher information this plays out as follows. The Fisher information is an expectation taken by treating the data as random, whereas the MLE you propose to substitute as the parameter will be the MLE from a particular fixed dataset. If you substitute the parameter $$\hat{\theta}_\text{MLE}$$ computed from some dataset $$\mathbf{x}$$ then the resulting Fisher information $$\mathcal{I}(\hat{\theta}_\text{MLE})$$ is still computed as an expectation that treats the data as random data generated from the model with the parameter $$\theta = \hat{\theta}_\text{MLE}$$. For most of the randomly generated values from this model, the score function will not be zero and so the parameter point $$\hat{\theta}_\text{MLE}$$ is not the MLE for that newly generated data.

Under mild regularity conditions, the expected value of the scores is zero. I.e., these are the first-order partial derivatives that have an expectation of zero (sum to zero) at the MLE. Thus, although:

$$\operatorname{E} \left[\left.{\frac{\partial}{\partial\theta}} \log f(X;\theta)\,\,\right|\,\,\theta \right] = 0,$$

the Fisher information is defined to be the variance of the score:

$$\mathcal{I}(\theta)= \int _{\mathbb {R} }\left({\frac {\partial }{\partial \theta }}\log f(x;\theta)\right)^{2}f(x;\theta )\,dx$$

Note how the square is taken, first, which explains why the Fisher information is generally not 0.

Given observed data and an MLE parameter estimate, the scores of observed data sum to zero at the MLE, but for the Fisher information the scores are squared before they are summed, in general yielding a non-zero value.

Another possible confusion comes from the distinction between the estimator $$\hat\theta$$ and the value it has in the current dataset.

Let's write $$\dot \ell(\theta)$$ for the score function. It's true by construction that $$\dot \ell(\hat\theta)=0$$ because that's how you get the MLE.

Since $$\dot \ell(\hat\theta)$$ is always zero it might seem that $$\mathrm{var}\left[\dot \ell(\hat\theta) \right]$$ must be zero and so can't be the Fisher information. The confusion here is between $$\hat\theta$$ the estimator, a random variable, and $$\hat\theta$$ the observed value, a constant.

Suppose $$\hat\theta=4$$. We must have $$\dot \ell(4)=0$$ in this data set, but in other data sets we typically won't have $$\dot \ell(4)=0$$. We can compute $$\mathrm{var}_{\theta=4}\left[\dot \ell(4) \right]$$ which will be some non-zero quantity. This quantity is not quite the Fisher information, because it's evaluated at $$\theta=4$$ rather than the true $$\theta=\theta_0$$, but in large samples it will be close to the true Fisher information because the estimated value of $$\theta$$ will be close to the true value.

When talk about the variance of the score at $$\hat\theta$$, that's what we mean. We take the value of $$\hat\theta$$ in this data set, treat this number as the true value, and compute the variance of $$\dot \ell(\theta)$$ when $$\theta$$ is fixed at this value.