32
$\begingroup$

Why is the test statistic of a likelihood ratio test distributed chi-squared?

$2(\ln \text{ L}_{\rm alt\ model} - \ln \text{ L}_{\rm null\ model} ) \sim \chi^{2}_{df_{\rm alt}-df_{\rm null}}$

$\endgroup$
  • 3
    $\begingroup$ Does this help? $\endgroup$ – Nick Sabbe Mar 20 '13 at 16:09
  • 14
    $\begingroup$ Thanks for the reference. Here's one from me: stats.stackexchange.com/faq#etiquette $\endgroup$ – Dr. Beeblebrox Mar 20 '13 at 16:50
  • 4
    $\begingroup$ Notice the "Bring your sense of humour" there. I did not intend to be rude, but the answer to this question would be relatively tedious and consist, basically, of the contents of that article (or some of the better statistics textbooks). If you state your precise problem with the explanation in one of these, I shall be glad to help you. $\endgroup$ – Nick Sabbe Mar 20 '13 at 17:17
23
$\begingroup$

As mentioned by @Nick this is a consequence of Wilks' theorem. But note that the test statistic is asymptotically $\chi^2$-distributed, not $\chi^2$-distributed.

I am very impressed by this theorem because it holds in a very wide context. Consider a statistical model with likelihood $l(\theta \mid y)$ where $y$ is the vector observations of $n$ independent replicated observations from a distribution with parameter $\theta$ belonging to a submanifold $B_1$ of $\mathbb{R}^d$ with dimension $\dim(B_1)=s$. Let $B_0 \subset B_1$ be a submanifold with dimension $\dim(B_0)=m$. Imagine you are interested in testing $H_0\colon\{\theta \in B_0\}$.

The likelihood ratio is $$lr(y) = \frac{\sup_{\theta \in B_1}l(\theta \mid y)}{\sup_{\theta \in B_0}l(\theta \mid y)}. $$ Define the deviance $d(y)=2 \log \big(lr(y)\big)$. Then Wilks' theorem says that, under usual regularity assumptions, $d(y)$ is asymptotically $\chi^2$-distributed with $s-m$ degrees of freedom when $H_0$ holds true.

It is proven in Wilk's original paper mentioned by @Nick. I think this paper is not easy to read. Wilks published a book later, perhaps with an easiest presentation of his theorem. A short heuristic proof is given in Williams' excellent book.

$\endgroup$
  • 3
    $\begingroup$ Sad that this theorem is not mentioned in the wikipedia page devoted to Samuel S. Wilks $\endgroup$ – Stéphane Laurent Mar 20 '13 at 17:33
  • 5
    $\begingroup$ Oh come on Stephane. This is Wikipedia, you can edit it and improve it! $\endgroup$ – StasK Mar 20 '13 at 19:19
  • 1
    $\begingroup$ @StasK I know that but I have never tried. And I already spend too much time of my life with statistics & mathematics ;) $\endgroup$ – Stéphane Laurent Mar 20 '13 at 19:23
  • $\begingroup$ Is there an intuition for why the 2 is in front of the log in the definition of the deviance? $\endgroup$ – user56834 Feb 13 '18 at 18:58
  • $\begingroup$ @Programmer2134 It is derived from a second order taylor expansion. $\endgroup$ – Frank Vel Sep 19 '18 at 12:50
25
$\begingroup$

I second Nick Sabbe's harsh comment, and my short answer is, It is not. I mean, it only is in the normal linear model. For absolutely any other sort of circumstances, the exact distribution is not a $\chi^2$. In many situations, you can hope that Wilks' theorem conditions are satisfied, and then asymptotically the log-likelihood ratio test statistics converges in distribution to $\chi^2$. Limitations and violations of the conditions of Wilks' theorem are too numerous to disregard.

  1. The theorem assumes i.i.d. data $\Rightarrow$ expect issues with dependent data, such as time series or unequal probability survey samples (for which the likelihoods are poorly defined, anyway; the "regular" $\chi^2$ tests, such as independence tests in contingency tables, start behaving as a sum $\sum_k a_k v_k, v_k \sim \mbox{i.i.d.} \chi^2_1$ (Rao & Scott). For i.i.d. data, $a_k=1$, and the sum becomes the $\chi^2$. But for non-independent data, this is no longer the case.
  2. The theorem assumes the true parameter to be in the interior of the parameter space. If you have a Euclidean space to work with, that's not an issue. However, in some problems, the natural restrictions may arise, such as variance $\ge$ 0 or correlation between -1 and 1. If the true parameter is one the boundary, then the asymptotic distribution is a mixture of $\chi^2$ with different degrees of freedom, in the sense that the cdf of the test is the sum of such cdfs (Andrews 2001, plus two or three more of his papers from the same period, with history going back to Chernoff 1954).
  3. The theorem assumes that all the relevant derivatives are non-zero. This can be challenged with some nonlinear problems and/or parameterizations, and/or situations when a parameter is not identified under the null. Suppose you have a Gaussian mixture model, and your null is one component $N(\mu_0,\sigma^2_0)$ vs. the alternative of two distinct components $f N(\mu_1,\sigma_1^2) + (1-f) N(\mu_2,\sigma_2^2)$ with a mixing fraction $f$. The null is apparently nested in the alternative, but this can be expressed in a variety of ways: as $f=0$ (in which case the parameters $\mu_1,\sigma_1^2$ are not identified), $f=1$ (in which case $\mu_2, \sigma_2^2$ are not identified), or $\mu_1=\mu_2, \sigma_1=\sigma_2$ (in which case $f$ is not identified). Here, you can't even say how many degrees of freedom your test should have, as you have different number of restrictions depending on how you parameterize the nesting. See the work of Jiahua Chen on this, e.g. CJS 2001.
  4. The $\chi^2$ may work OK if the distribution has been correctly specified. But if it was not, the test will break down again. In the (largely neglected by statisticians) subarea of multivariate analysis known as structural equation covariance modeling, a multivariate normal distribution is often assumed, but even if the structure is correct, the test will misbehave if the distribution is different. Satorra and Bentler 1995 show that the distribution will become $\sum_k a_k v_k, v_k \sim \mbox{i.i.d.} \chi^2_1$, the same story as with non-independent data in my point 1, but they've also demonstrated how the $a_k$s depend on the structure of the model and the fourth moments of the distribution.
  5. For finite samples, in a large class of situations likelihood ratio is Bartlett-correctible: while ${\rm Prob}[d(y) \le x]=F(x;\chi^2_d)[1+O(n^{-1})]$ for a sample of size $n$, and $F(x;\chi^2_d)$ being the distribution function of the $\chi^2_d$ distribution, for the regular likelihood problems you can find a constant $b$ such that ${\rm Prob}[d(y)/(1+b/n) \le x]=F(x;\chi^2_d)[1+O(n^{-2})]$, i.e., to a higher order of accuracy. So the $\chi^2$ approximation for finite samples can be improved (and arguably should be improved if you know how). The constant $b$ depends on the structure of the model, and sometimes on the auxiliary parameters, but if it can be consistently estimated, that works, too, in improving the order of coverage.

For a review of these and similar esoteric issues in likelihood inference, see Smith 1989.

$\endgroup$
  • 1
    $\begingroup$ Thanks! Very instructive. What do you mean by "it only is in the normal linear model" ? For a Fisher test when $B_0$ and $B_1$ are linear subspaces, then the deviance is a monotone function of the Fisher statistic, and it is only asymptotically $\chi^2$. $\endgroup$ – Stéphane Laurent Mar 20 '13 at 19:46
  • $\begingroup$ With known variance, I should add. $\endgroup$ – StasK Mar 20 '13 at 19:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.