2
$\begingroup$

In the book Mathematical Methods for Physics and Engineering it is said that the likelihood function tends to a Gaussian (centred on the maximum-likelihood estimate) in the large sample limit. The way it is phrased makes it seem like they are saying this is due to the central limit theorem, but I am struggling to see how it is relevant. It relies on the random variable being a sum of a sequence of other random variables, which I don't think is the case here.

I believe this is often misunderstood, for example in this question, which I have several problems with. The arguments use the central limit theorem to find the distribution of the likelihood and show that it is asymptotically normal. However, we are not interested in its distribution as a random variable; we instead care about its functional form as the parameters are varied for given observed sample values.

As an example of what I mean, suppose we draw $n$ sample values $x_i$ from a distribution $P(x|\tau)=(1/\tau)\exp(-x/\tau)$. The likelihood function is then $$L(\boldsymbol{x};\tau)=P(x_1|\tau)P(x_2|\tau)\dots P(x_n|\tau)=\frac{1}{\tau^n}\exp{\left[-\frac{\sum_i x_i}{\tau}\right]}.$$ Suppose we now evaluate this using the observed values of $x_i$ and consider it as a function of $\tau$. In general this will obviously be different every time, but the book says that in the limit $n\to\infty$, the function tends to a Gaussian with peak centred on the maximum likelihood estimate $\hat{\tau}$ and width inversely proportional $\sqrt{n}$. Why should we expect this to be the case?

$\endgroup$
15
  • 1
    $\begingroup$ Does this answer your question? Proof that ML Estimator is asymptotically Normal $\endgroup$ Commented Apr 13, 2023 at 12:33
  • 1
    $\begingroup$ I think this is different, as it is the maximum likelihood estimator rather than the likelihood function $\endgroup$ Commented Apr 13, 2023 at 12:46
  • 4
    $\begingroup$ The log likelihood (not the likelihood itself!) is a sum of iid random variables. Throw in the usual technical requirements (e.g., finite variances) and QED. $\endgroup$
    – whuber
    Commented Apr 13, 2023 at 13:30
  • 1
    $\begingroup$ The distribution of a properly-normalized sum will approach a Gaussian (with a few extra conditions...) but the distribution of the sum is not itself Gaussian (unless the variables being summed are all Gaussian.) $\endgroup$
    – jbowman
    Commented Apr 13, 2023 at 20:26
  • 1
    $\begingroup$ I'm sorry, but the phrase "distribution of the sum is Gaussian, not the sum itself" makes no sense to me: in what sense is a sum of random variables "Gaussian" if not in terms of its distribution?? Regardless, the CLT does not assert any distribution of any sum is Gaussian: it's a limiting statement about distributions of sums. See our posts on the CLT for more about that. We care very much about the distribution of the log likelihood because that tells us how certain we can be about estimates. $\endgroup$
    – whuber
    Commented Apr 13, 2023 at 20:28

1 Answer 1

3
$\begingroup$

I think the question here is why/whether the loglikelihood function is asymptotically locally a parabola with its maximum at $\hat\theta$ (and so the likelihood function is locally $\exp(-(\theta-\hat\theta)^2/2)$.Basically, it's just Taylor series: any smooth function is locally a parabola, and any smooth function with zero derivative at a point is locally a parabola with a max or min at that point. It isn't quite just Taylor series -- we do need the Law of Large Numbers and also the rate of convergence of an average to its expected value to be sure that the terms in the Taylor expansion are of the sizes we need them to be.

In this case (and relevant other cases) the log-likelihood function is smooth. By the definition of the MLE, the loglikelihood has a maximum at $\hat\theta$ and the derivative is zero at $\hat\theta$. Therefore, $$\ell(\theta)=\ell(\hat\theta) + 0\times (\theta-\hat\theta) + \ell''(\hat\theta)(\theta-\hat\theta)^2 +\text{remainder}$$ Since the derivatives of $\ell$ are proportional to $n$, we need $\theta-\hat\theta= O_p(n^{-1/2}$ to keep the remainder small, but that's the range we're interested in. On that range, $\ell(\hat\theta)-\ell(\theta)$ is asymptotically a parabola centered at $\hat\theta$, so $\exp(\ell(\theta))$ is asymptotically the same shape as a Gaussian likelihood function.

$\endgroup$
2
  • $\begingroup$ Great, thanks. So, am I right in saying that this seems quite different to the central limit theorem? $\endgroup$ Commented Apr 14, 2023 at 10:09
  • $\begingroup$ Yes, it does seem quite different. $\endgroup$ Commented Apr 15, 2023 at 3:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.