Does asymptotic normality of the likelihood function follow from the central limit theorem?

Question

In the book Mathematical Methods for Physics and Engineering it is said that the likelihood function tends to a Gaussian (centred on the maximum-likelihood estimate) in the large sample limit. The way it is phrased makes it seem like they are saying this is due to the central limit theorem, but I am struggling to see how it is relevant. It relies on the random variable being a sum of a sequence of other random variables, which I don't think is the case here.

I believe this is often misunderstood, for example in this question, which I have several problems with. The arguments use the central limit theorem to find the distribution of the likelihood and show that it is asymptotically normal. However, we are not interested in its distribution as a random variable; we instead care about its functional form as the parameters are varied for given observed sample values.

As an example of what I mean, suppose we draw $n$ sample values $x_i$ from a distribution $P(x|\tau)=(1/\tau)\exp(-x/\tau)$. The likelihood function is then $$L(\boldsymbol{x};\tau)=P(x_1|\tau)P(x_2|\tau)\dots P(x_n|\tau)=\frac{1}{\tau^n}\exp{\left[-\frac{\sum_i x_i}{\tau}\right]}.$$ Suppose we now evaluate this using the observed values of $x_i$ and consider it as a function of $\tau$. In general this will obviously be different every time, but the book says that in the limit $n\to\infty$, the function tends to a Gaussian with peak centred on the maximum likelihood estimate $\hat{\tau}$ and width inversely proportional $\sqrt{n}$. Why should we expect this to be the case?

Answer 1

I think the question here is why/whether the loglikelihood function is asymptotically locally a parabola with its maximum at $\hat\theta$ (and so the likelihood function is locally $\exp(-(\theta-\hat\theta)^2/2)$.Basically, it's just Taylor series: any smooth function is locally a parabola, and any smooth function with zero derivative at a point is locally a parabola with a max or min at that point. It isn't quite just Taylor series -- we do need the Law of Large Numbers and also the rate of convergence of an average to its expected value to be sure that the terms in the Taylor expansion are of the sizes we need them to be.

In this case (and relevant other cases) the log-likelihood function is smooth. By the definition of the MLE, the loglikelihood has a maximum at $\hat\theta$ and the derivative is zero at $\hat\theta$. Therefore, $$\ell(\theta)=\ell(\hat\theta) + 0\times (\theta-\hat\theta) + \ell''(\hat\theta)(\theta-\hat\theta)^2 +\text{remainder}$$ Since the derivatives of $\ell$ are proportional to $n$, we need $\theta-\hat\theta= O_p(n^{-1/2}$ to keep the remainder small, but that's the range we're interested in. On that range, $\ell(\hat\theta)-\ell(\theta)$ is asymptotically a parabola centered at $\hat\theta$, so $\exp(\ell(\theta))$ is asymptotically the same shape as a Gaussian likelihood function.