Does the deviance have a known sampling distribution or not?

Question

In their book Generalized Linear Models, McCullagh & Nelder seem to imply that the sampling distribution of the deviance is generally not known:

This is strange, because the deviance of a model $m$ is defined as $$ D = -2 \bigl(\ell_m(\hat\theta_\text{ML}; x) - \ell_s(\hat\theta_\text{ML}; x) \bigr) $$ where $s$ denotes the so-called "saturated" model. In other words, the deviance is simply the test statistic from the likelihood-ratio test for two nested models, which, as far as I know, has a well-known (asymptotic) distribution. From the Wikipedia:

A convenient result, attributed to Samuel S. Wilks, says that as the sample size $n$ approaches $\infty$, the test statistic $-2 \log(\Lambda)$ for a nested model will be asymptotically $\chi^2$-distributed with degrees of freedom equal to the difference in dimensionality of $\Theta$ and $\Theta_0$.

So, do we know the distribution of $D$, yes or no?

Answer 1

A better idea is that instead of thinking about the sampling distribution you use Posterior samples $\tilde{y}$ using MCMC P($\tilde{y}|y)$ = $ \int_\theta P(\tilde{y}|\theta)P(\theta|y)d\theta$ to generate a bunch of Deviance samples $D = -2*\sum_ilog(P(\tilde{y}))$. MCMC methods allow you to generate posterior samples. Since posterior samples encapsulate the uncertainty it is propagated to the deviance and that is what you really want when you make decisions. You use measures like WAIC and LOO which average deviances in very specific ways that are derived from solid theory based in Cross Validation.

A really good computational reference is McElreath's new book's Chapter 6 and for theory on this papers like this and this one that explain these concepts.

Of course posterior estimation are not easy or possible in many models. But for most Linear, Generalized Linear(especially these) models these methods are easily applicable(Basically anywhere where you have a maximum likelihood that's well defined)