Why is a likelihood-ratio test distributed chi-squared?

Question

Why is the test statistic of a likelihood ratio test distributed chi-squared?

$2(\ln \text{ L}_{\rm alt\ model} - \ln \text{ L}_{\rm null\ model} ) \sim \chi^{2}_{df_{\rm alt}-df_{\rm null}}$

Answer 1

As mentioned by @Nick this is a consequence of Wilks' theorem. But note that the test statistic is asymptotically $\chi^2$-distributed, not $\chi^2$-distributed.

I am very impressed by this theorem because it holds in a very wide context. Consider a statistical model with likelihood $l(\theta \mid y)$ where $y$ is the vector observations of $n$ independent replicated observations from a distribution with parameter $\theta$ belonging to a submanifold $B_1$ of $\mathbb{R}^d$ with dimension $\dim(B_1)=s$. Let $B_0 \subset B_1$ be a submanifold with dimension $\dim(B_0)=m$. Imagine you are interested in testing $H_0\colon\{\theta \in B_0\}$.

The likelihood ratio is $$lr(y) = \frac{\sup_{\theta \in B_1}l(\theta \mid y)}{\sup_{\theta \in B_0}l(\theta \mid y)}. $$ Define the deviance $d(y)=2 \log \big(lr(y)\big)$. Then Wilks' theorem says that, under usual regularity assumptions, $d(y)$ is asymptotically $\chi^2$-distributed with $s-m$ degrees of freedom when $H_0$ holds true.

It is proven in Wilk's original paper mentioned by @Nick. I think this paper is not easy to read. Wilks published a book later, perhaps with an easiest presentation of his theorem. A short heuristic proof is given in Williams' excellent book.

Answer 2

I second Nick Sabbe's harsh comment, and my short answer is, It is not. I mean, it only is in the normal linear model. For absolutely any other sort of circumstances, the exact distribution is not a $\chi^2$. In many situations, you can hope that Wilks' theorem conditions are satisfied, and then asymptotically the log-likelihood ratio test statistics converges in distribution to $\chi^2$. Limitations and violations of the conditions of Wilks' theorem are too numerous to disregard.

1. The theorem assumes i.i.d. data $\Rightarrow$ expect issues with dependent data, such as time series or unequal probability survey samples (for which the likelihoods are poorly defined, anyway; the "regular" $\chi^2$ tests, such as independence tests in contingency tables, start behaving as a sum $\sum_k a_k v_k, v_k \sim \mbox{i.i.d.} \chi^2_1$ (Rao & Scott). For i.i.d. data, $a_k=1$, and the sum becomes the $\chi^2$. But for non-independent data, this is no longer the case.
2. The theorem assumes the true parameter to be in the interior of the parameter space. If you have a Euclidean space to work with, that's not an issue. However, in some problems, the natural restrictions may arise, such as variance $\ge$ 0 or correlation between -1 and 1. If the true parameter is one the boundary, then the asymptotic distribution is a mixture of $\chi^2$ with different degrees of freedom, in the sense that the cdf of the test is the sum of such cdfs (Andrews 2001, plus two or three more of his papers from the same period, with history going back to Chernoff 1954).
3. The theorem assumes that all the relevant derivatives are non-zero. This can be challenged with some nonlinear problems and/or parameterizations, and/or situations when a parameter is not identified under the null. Suppose you have a Gaussian mixture model, and your null is one component $N(\mu_0,\sigma^2_0)$ vs. the alternative of two distinct components $f N(\mu_1,\sigma_1^2) + (1-f) N(\mu_2,\sigma_2^2)$ with a mixing fraction $f$. The null is apparently nested in the alternative, but this can be expressed in a variety of ways: as $f=0$ (in which case the parameters $\mu_1,\sigma_1^2$ are not identified), $f=1$ (in which case $\mu_2, \sigma_2^2$ are not identified), or $\mu_1=\mu_2, \sigma_1=\sigma_2$ (in which case $f$ is not identified). Here, you can't even say how many degrees of freedom your test should have, as you have different number of restrictions depending on how you parameterize the nesting. See the work of Jiahua Chen on this, e.g. CJS 2001.
4. The $\chi^2$ may work OK if the distribution has been correctly specified. But if it was not, the test will break down again. In the (largely neglected by statisticians) subarea of multivariate analysis known as structural equation covariance modeling, a multivariate normal distribution is often assumed, but even if the structure is correct, the test will misbehave if the distribution is different. Satorra and Bentler 1995 show that the distribution will become $\sum_k a_k v_k, v_k \sim \mbox{i.i.d.} \chi^2_1$, the same story as with non-independent data in my point 1, but they've also demonstrated how the $a_k$s depend on the structure of the model and the fourth moments of the distribution.
5. For finite samples, in a large class of situations likelihood ratio is Bartlett-correctible: while ${\rm Prob}[d(y) \le x]=F(x;\chi^2_d)[1+O(n^{-1})]$ for a sample of size $n$, and $F(x;\chi^2_d)$ being the distribution function of the $\chi^2_d$ distribution, for the regular likelihood problems you can find a constant $b$ such that ${\rm Prob}[d(y)/(1+b/n) \le x]=F(x;\chi^2_d)[1+O(n^{-2})]$, i.e., to a higher order of accuracy. So the $\chi^2$ approximation for finite samples can be improved (and arguably should be improved if you know how). The constant $b$ depends on the structure of the model, and sometimes on the auxiliary parameters, but if it can be consistently estimated, that works, too, in improving the order of coverage.

For a review of these and similar esoteric issues in likelihood inference, see Smith 1989.

Answer 3

As other commentators have pointed out, Wilks' theorem (Wilks 1938) only shows that, under various regularity conditions, this statistic is asymptotically chi-squared distributed. The asymptotic result follows from taking a multivariate Taylor expansion of the log-likelihood function and looking at what happens when the MLE is a critical point of the function. Using various asymptotic results relating to the MLE it is possible to eliminate all terms in the expansion except for the second-order term, which turns asymptotically into the squared norm of a normal random vector.

Derivations of Wilks' theorem can be found in various textbooks on estimation theory, and there are also versions floating around in online statistics lecture notes (see e.g., here). The general derivation requires a knowledge of mutivariate Taylor series and results pertaining to the MLE of a vector parameter. A simpler version of the derivation can be shown in the scalar case where the alternative model has only one more (scalar) parameter than the null model. For illustrative purposes, I will show the heuristic derivation of the result in this case.

---

Heuristic demonstration of Wilks' theorem with one degree-of-freedom: Consider the simple case where we have an alternative hypothesis with only one scalar parameter $\theta$ that is fixed to the value $\theta_0$ under the null hypothesis. In this case we have ${df}_A - {df}_0 = 1$ so the asymptotic distribution is a chi-squared distribution with one degree-of-freedom. To derive this asymptotic distribution we will use the following Taylor expansion:

$$\ell_\mathbf{x}(\theta_0) = \ell_\mathbf{x}(\hat{\theta}_n) + \ell_\mathbf{x}'(\hat{\theta}_n) (\theta_0 - \hat{\theta}_n) + \frac{\ell_\mathbf{x}''(\hat{\theta}_n)}{2} (\theta_0 - \hat{\theta}_n)^2 + \mathcal{O}((\theta_0 - \hat{\theta}_n)^3).$$

To facilitate our analysis, we define the standardised estimation error $E_n(\theta) \equiv (\theta - \hat{\theta}_n) \sqrt{n\mathcal{I}(\theta)}$ where $\mathcal{I}$ is the Fisher information function. Now, suppose that the MLE $\hat{\theta}_n$ occurs at a critical point of the log-likelihood function so that $\ell_\mathbf{x}'(\hat{\theta}_n) = 0$. This gives the following simplified form for the Taylor expansion:

$$\begin{aligned} \ell_\mathbf{x}(\theta_0) &= \ell_\mathbf{x}(\hat{\theta}_n) + \frac{\ell_\mathbf{x}''(\hat{\theta}_n)}{2} (\theta_0 - \hat{\theta}_n)^2 + \mathcal{O}((\theta_0 - \hat{\theta}_n)^3) \\[6pt] &= \ell_\mathbf{x}(\hat{\theta}_n) + \frac{\ell_\mathbf{x}''(\hat{\theta}_n)}{2 n \mathcal{I}(\theta_0)} \cdot E_n(\theta_0)^2 + \mathcal{O} \bigg( \frac{E_n(\theta_0)^3}{n^{3/2}} \bigg). \\[6pt] \end{aligned}$$

Substituting this expansion into the likelihood-ratio statistic we get:

$$\begin{aligned} W_n &\equiv 2(\ell_\mathbf{x}(\hat{\theta}_n) - \ell_\mathbf{x}(\theta_0)) \\[6pt] &= - \frac{\ell_\mathbf{x}''(\hat{\theta}_n)}{n \mathcal{I}(\theta_0)} \cdot E_n(\theta_0)^2 + \mathcal{O} \bigg( \frac{E_n(\theta_0)^3}{n^{3/2}} \bigg). \\[6pt] \end{aligned}$$

Now, suppose you are looking at the distribution of $W_n$ under the null hypothesis that $\theta = \theta_0$. Under some regularity conditions, it is known that we get the asymptotic distribution $E_n(\theta_0) \sim \text{N}(0, 1)$ and we also get the limiting result $\ell_\mathbf{x}''(\hat{\theta}_n)/n \rightarrow -\mathcal{I}(\theta_0)$. This means that the order term in the above expansion will vanish asymptotically, and so we have the asymptotic result:

$$\begin{aligned} W_n \rightarrow E_n(\theta_0)^2 \sim \chi_{1}^2. \\[6pt] \end{aligned}$$

This is the chi-squared asymptotic result that holds in the case where the alternative model has only one more degree-of-freedom than the null model. The more general derivation is essentially the same, but it involves use of a multivariate parameter vector, which means we use the multivariate Taylor series and the properties of the MLE for a vector parameter.

---

As others have noted, Wilks' theorem uses a number of regularity conditions, and these conditions do not always hold. The result assumes that the MLE occurs at an interior point of the parameter space which is a critical point of the log-likelihood function. Additionally, it assumes all the required conditions for the standard asymptotic normality results for the MLE. Even when these various regularity conditions hold (which actually happens in quite a broad range of cases), the result is only an asymptotic result, and so it might not be a particularly good approximation for small sample sizes.