Asymptotic chi-squared distribution of likelihood ratio statistic in regression problem

Question

There is a famous result, going back to Wilks (1938) "The large-sample distribution of the likelihood ratio for testing composite hypotheses" (Ann. Math. Stat., 9, 60-62) that states that if we compare two nested models with dimensions $p<q$ and compute the likelihood ratio $\lambda$, then $-2\log\lambda$ is asymptotically distributed $\chi^2_{q-p}$, assuming as null hypothesis the lower dimensional model. This assumes that the ML-estimators are consistent and asymptotically normal. It also assumes the data to be i.i.d. Despite this, $-2\log\lambda$ is routinely taken to be asymptotically $\chi^2_{q-p}$ also in regression problems where normally the explanatory variables ${\bf X}$ are assumed fixed and the response $y$ random. Then observations are no longer i.i.d. because for observation $(y_i,{\bf X}_i)$ the distribution of $y_i$ depends on ${\bf X}_i$.

I haven't managed to find a rigorous proof including required conditions for this result in the regression case, and this is what I'm asking for. Does anybody know where such a proof is given? Note that my personal interest is in a generalised linear model, but it may be of help to see if such a result exists for other regression problems as well, for example linear regression with non-normal errors, or specific nonlinear regression models.

Some findings and thoughts (I add to them as I'm finding more):

(1) Looking at Wilks' argument I wonder whether one could argue that this just holds without assuming i.i.d., but assuming asymptotic normality of the ML-estimators alone; it doesn't seem to be required anywhere that the density $f$ is the same for all observations (Wilks' paper is not fully rigorous though). However I haven't found this stated explicitly anywhere. And (2) and (3) below somehow suggest that it isn't that easy, because otherwise those cases wouldn't have been worthwhile to mention!?

(2) I have seen a result for the non-i.i.d. case in which there are assumed groups of $n_1,\ n_2,\ldots,n_m$ observations with different densities. This result however requires that the number of groups is fixed and all $n_j$ go to infinity. This seems too strong for regression problems.

(3) Van der Vaart's book "Asymptotic Statistics" has the distribution of the likelihood ratio statistic in generalised linear models as Example 16.8. However, he sets this up as i.i.d. problem with random ${\bf X}$ and assumes that $(y,{\bf X})$ are jointly distributed according to an exponential family. I would like to avoid this assumption and prefer fixed ${\bf X}$. (Elsewhere I found a result assuming ${\bf X}$ to be normally distributed; again this doesn't make me happy enough.)

(4) I believe (although I haven't checked it because it is not of immediate use to me) that the result will follow pretty easily for a standard linear regression model with normal errors and assumptions on ${\bf X}$ that grant good asymptotic behaviour (basically collapsing to degeneracy needs to be forbidden). Everything then is normal and nice and $-2\log \lambda$ has nothing but squares of normal things. But this will not carry over to the generalised linear model I'm interested in.

(5) I'd be fine to assume that the ML-estimators are asymptotically normal. In the situation I'm interested in, this is secured. I'm quite sure that something like this is needed (it features explicitly in Wilks' paper), so I'm not looking for conditions for this to hold, rather for a proof telling me whether the chi-squared distribution of $-2\log\lambda$ needs only that or what more.

(6) The "Introduction to Generalized Linear Models" (2nd ed.) by A. J. Dobson has an explicit claim in Sec. 5.5/5.6 that does not assume the ${\bf X}$ to be random, and also gives something like a "proof", based (as usually) on the asymptotic normality of the ML-estimator. This is not rigorous though, in that there is no discussion of whether remainder terms really vanish or that conditions for this would be given; it basically states that some equations hold "approximately" without being precise. Also no reference to a rigorous proof.

Answer 1

For the $\chi^2$ distribution to hold needs more than asymptotic Normality. The condition is basically that $$\sqrt{n}(\hat\beta-\beta)\stackrel{d}{\to} N(0,I^{-1})$$ where $-I$ is the average expected second derivative of the log likelihood. Suppose, more generally, that $$\sqrt{n}(\hat\beta-\beta)\stackrel{d}{\to} N(0,V)$$ and look at the test for all the parameters rather than just a subset of them, to simplify things.

Let $\ell(\beta)$ be the loglikelihood and $\beta_0$ the null hypothesis value. Assume we can do a second-order Taylor expansion

$$\ell(\beta_0)= \ell(\hat\beta)+ (\hat\beta-\beta_0)\ell'(\hat\beta)+\frac{1}{2}(\hat\beta-\beta_0)^t\ell''(\hat\beta)(\hat\beta-\beta_0)+o_p(\|\hat\beta-\beta_0\|^2)$$

The second term on the RHS is zero because $\hat\beta$ is the maximum, so $Q=-(2\ell(\hat\beta)-2\ell(\beta_0))$ is asymptotically a quadratic form in $\hat\beta-\beta_0$. Note now that $\ell''(\hat\beta)=-nI(\hat\beta)$ (by definition). Asymptotic normality of $\hat\beta$ now implies that the asymptotic distribution of $Q$ is a quadratic form in Gaussians; the distribution of $$\sum_{i=1}^p\lambda_iZ_i^2$$ where $Z_i$ are iid $N(0,1)$ and $\lambda_i$ are the eigenvalues of $IV$. If $V$ is obtained by maximimising $\ell$, then $V=I^{-1}JI^{-1}$, where $J$ is the variance of $n^{-1/2}\ell'(\beta_0)$, so $IV=JI^{-1}$

The distribution will be a $\chi^2$ distribution if the eigenvalues are all 0 or 1, otherwise it won't. If we're looking at the test for all parameters, this means $JI^{-1}$ must be the identity, and so $V=I^{-1}$. When testing just some parameters the same arguments as usual extend to show you still want $V=I^{-1}$.

In many situations it's fine for the distribution to be a more general quadratic form in Gaussians, since $\lambda_i$ can be estimated and there are several algorithms for computing the tail probabilities of the linear combination of $Z^2$. In particular, in survey statistics these tests are called the Rao-Scott likelihood ratio tests (especially in the case of loglinear models for contingency tables), and the tail probabilities are typically approximated using the Satterthwaite approximation.