# Is the Quadratic Approximation of Log-Likelihood Equivalent to the Normal Approximation of the MLE?

Let $$X_1, X_2, ..., X_n \sim \text{IID N}(\theta, \sigma^2)$$ with $$\sigma^2$$ known, and let $$\hat{\theta}$$ be the MLE of the mean.

(1) How can I show that in this case, the following is true?

$$\log \frac{L(\theta)}{L(\hat{\theta})} = - \frac{1}{2} I(\hat{\theta}) (\theta - \hat{\theta})^2.$$

As I understand it, part of the proof would need to show that the following term in the Taylor expansion of the likelihood function about the MLE is zero:

$$S(\hat{\theta}) (\theta - \hat{\theta}).$$

If I read it correctly, Yudi Pawitan, suggests that the above term is exactly equal to zero when the $$X$$ values are exactly Normally Distributed.

(2) Is the above term equal to zero if the x's are exactly normally distributed?

But isn't the score function evaluated at the MLE always zero, because that is how we found the MLE in the first place, by solving for the MLE that equates the score function to zero?

(3) Does the equation (1) hold only if the $$X$$ values are exactly normally distributed? Or, equivalently, is the term in (2) equal to zero only if the $$X$$ values are normally distributed? Why doesn't (1) hold if the $$X$$ values have a distribution other than (exactly) normal? Why isn't (2) exactly zero regardless of the distribution of the $$X$$ values?

I know that I am missing something fundamental here, so hoping someone could enlighten me.

• This is discussed here – kjetil b halvorsen Dec 22 '18 at 21:56
• @kjetil: Thank you for the reference. The following sentence in which you quote Charles J. Geyer is the one I am still trying to understand at some level "If the log likelihood is approximately quadratic with constant Hessian, then the maximum likelihood estimator (MLE) is approximately normally distributed. No other assumptions are required." – ColorStatistics Dec 24 '18 at 6:59

How can I show this is the case?

Since you have full specification of the sampling distribution of your observations, you can get the explicit form of the log-likelihood. Treating $$\sigma$$ as fixed and removing additive constants we have:

$$\ell_\mathbf{x}(\theta) = -\frac{1}{2 \sigma^2} \sum_{i=1}^n (x_i - \theta)^2 \quad \quad \quad \text{for all } \theta \in \mathbb{R}.$$

From this function it is possible to derive the score function, the information function and the MLE, which means that you should be able to directly verify the equation by substituting all these items. (I will leave this work as an exercise.)

Isn't the score of the MLE always zero?

To understand when the score of the MLE is zero, think back to your early calculus classes. When you maximise a continuous differentiable function, this often gives a maximising value at a critical point of the function. But the maximising value is not always at a critical point. In some cases it may be at a boundary point of the function. Now, in the context of maximum-likelihood, it is common for the log-likelihood function to be strictly concave, so that there is a unique MLE at the critical point of the function --- i.e., when the score function equals zero. However, we still need to be careful that this is the case, and it is possible in some cases that the MLE will occur at a boundary point. Remember that there is nothing special about maximum likelihood analysis --- mathematically it is just a standard optimisation problem involving a log-likelihood function, and it is solved via ordinary optimisation techniques.

Now, in this particular case, it turns out that the above log-likelihood function is strictly concave (show this by looking at its second-derivative) and so the MLE occurs at the unique critical point of the function. Thus, in this case, it is indeed correct that we find the MLE by setting the score function to zero (and so obviously the score of the MLE is equal to zero in this case).

When statisticians deal with maximum-likelihood theory, they often assume "regularity conditions" which are the conditions required to allow the log-likelihood to be expanded into a Taylor expansion, and to ensure that the MLE falls at a critical point. So if you read material on the properties of MLEs, you will often find that they are of the form, "Under such-and-such regularity conditions, such-and-such a result occurs".

Do these results depend on the data actually being normally distributed?

In these kinds of problems, the log-likelihood function is taken to be the derived from the distribution we think the data follows. So even if the distribution of the data turns out not to be normal, the context of the problem suggest that we think it is normal, so this is the log-likelihood function we use for our analysis. Similarly, we derive the MLE as if the data were normal, even if they turn out not to be.

In this particular case, all of the relevant equations you have should follow directly from the assumed form of the log-likelihood function, for all possible outcomes of the data. However, it is important to remember that the MLE is a function of the data, and so its probabilistic behaviour depends on the true distribution of the data, which might not be our assumed form. Thus, if you were to make some probabilistic statement about the MLE (e.g., that it will fall within a certain interval with a certain probability) then this would generally depend on the behaviour of the data, which would depend on its true distribution.

• Thank you very much, Ben. A major thing I was missing was that S(theta hat) need not be equal to zero in case of a boundary point maximum. Thank you for that point. – ColorStatistics Dec 24 '18 at 6:47
• One last connection I am trying to make, pertains to that between the regularity of log-likelihood and the normality of theta hat (not of the x's). Yudi Pawitan writes "a quadratic approximation of the log-likelihood corresponds to a normal approximation of theta hat." He seems to be saying that if we assume that the log-likelihood function is regular, or well approximated by a quadratic function, that is equivalent in some way to a normal approximation of theta hat. I don't understand what the "normal approximation of theta hat" refers to and how's it is connected to regularity. – ColorStatistics Dec 24 '18 at 6:50
• Got a bit more clarity here: Yudi Pawitan is saying is that if we assume that the ML estimate has a normal sampling distribution, which I read to mean that had we had drawn a different sample we would have obtained a different ML estimate and had we continued to do so for all samples of size n, we'd obtain the entire collection of ML estimates, which we assume to have a normal probability density. He then makes the puzzling connection between this normality assumption and the regularity of the log-likelihood function. – ColorStatistics Dec 24 '18 at 16:09
• Observe that the log-likelihood function above is a quadratic in $\theta$. This is what you will get if you take a quadratic approximation to the log-likelihood, so once you have done that, all the remaining results, including distribution of the MLE, all follow. It is also worth looking at some proofs of the asymptotic normality of the MLE in the general case. These proofs are usually based on showing that (under mild regularity conditions) you can take Taylor expansion of the log-likelihood, and the higher-order terms disappear for large $n$. – Ben - Reinstate Monica Dec 24 '18 at 21:18
• Thank you, Ben. I believe that answers it all. I'll work through the math to confirm to myself. – ColorStatistics Dec 25 '18 at 14:03