Is the Quadratic Approximation of Log-Likelihood Equivalent to the Normal Approximation of the MLE?

Question

Let $X_1, X_2, ..., X_n \sim \text{IID N}(\theta, \sigma^2)$ with $\sigma^2$ known, and let $\hat{\theta}$ be the MLE of the mean.

(1) How can I show that in this case, the following is true?

$$\log \frac{L(\theta)}{L(\hat{\theta})} = - \frac{1}{2} I(\hat{\theta}) (\theta - \hat{\theta})^2.$$

As I understand it, part of the proof would need to show that the following term in the Taylor expansion of the likelihood function about the MLE is zero:

$$S(\hat{\theta}) (\theta - \hat{\theta}).$$

If I read it correctly, Yudi Pawitan, suggests that the above term is exactly equal to zero when the $X$ values are exactly Normally Distributed.

(2) Is the above term equal to zero if the x's are exactly normally distributed?

But isn't the score function evaluated at the MLE always zero, because that is how we found the MLE in the first place, by solving for the MLE that equates the score function to zero?

(3) Does the equation (1) hold only if the $X$ values are exactly normally distributed? Or, equivalently, is the term in (2) equal to zero only if the $X$ values are normally distributed? Why doesn't (1) hold if the $X$ values have a distribution other than (exactly) normal? Why isn't (2) exactly zero regardless of the distribution of the $X$ values?

I know that I am missing something fundamental here, so hoping someone could enlighten me.

Answer 1

How can I show this is the case?

Since you have full specification of the sampling distribution of your observations, you can get the explicit form of the log-likelihood. Treating $\sigma$ as fixed and removing additive constants we have:

$$\ell_\mathbf{x}(\theta) = -\frac{1}{2 \sigma^2} \sum_{i=1}^n (x_i - \theta)^2 \quad \quad \quad \text{for all } \theta \in \mathbb{R}.$$

From this function it is possible to derive the score function, the information function and the MLE, which means that you should be able to directly verify the equation by substituting all these items. (I will leave this work as an exercise.)

Isn't the score of the MLE always zero?

To understand when the score of the MLE is zero, think back to your early calculus classes. When you maximise a continuous differentiable function, this often gives a maximising value at a critical point of the function. But the maximising value is not always at a critical point. In some cases it may be at a boundary point of the function. Now, in the context of maximum-likelihood, it is common for the log-likelihood function to be strictly concave, so that there is a unique MLE at the critical point of the function --- i.e., when the score function equals zero. However, we still need to be careful that this is the case, and it is possible in some cases that the MLE will occur at a boundary point. Remember that there is nothing special about maximum likelihood analysis --- mathematically it is just a standard optimisation problem involving a log-likelihood function, and it is solved via ordinary optimisation techniques.

Now, in this particular case, it turns out that the above log-likelihood function is strictly concave (show this by looking at its second-derivative) and so the MLE occurs at the unique critical point of the function. Thus, in this case, it is indeed correct that we find the MLE by setting the score function to zero (and so obviously the score of the MLE is equal to zero in this case).

When statisticians deal with maximum-likelihood theory, they often assume "regularity conditions" which are the conditions required to allow the log-likelihood to be expanded into a Taylor expansion, and to ensure that the MLE falls at a critical point. So if you read material on the properties of MLEs, you will often find that they are of the form, "Under such-and-such regularity conditions, such-and-such a result occurs".

Do these results depend on the data actually being normally distributed?

In these kinds of problems, the log-likelihood function is taken to be the derived from the distribution we think the data follows. So even if the distribution of the data turns out not to be normal, the context of the problem suggest that we think it is normal, so this is the log-likelihood function we use for our analysis. Similarly, we derive the MLE as if the data were normal, even if they turn out not to be.

In this particular case, all of the relevant equations you have should follow directly from the assumed form of the log-likelihood function, for all possible outcomes of the data. However, it is important to remember that the MLE is a function of the data, and so its probabilistic behaviour depends on the true distribution of the data, which might not be our assumed form. Thus, if you were to make some probabilistic statement about the MLE (e.g., that it will fall within a certain interval with a certain probability) then this would generally depend on the behaviour of the data, which would depend on its true distribution.