# Does the deviance have a known sampling distribution or not?

In their book Generalized Linear Models, McCullagh & Nelder seem to imply that the sampling distribution of the deviance is generally not known:

This is strange, because the deviance of a model $m$ is defined as $$D = -2 \bigl(\ell_m(\hat\theta_\text{ML}; x) - \ell_s(\hat\theta_\text{ML}; x) \bigr)$$ where $s$ denotes the so-called "saturated" model. In other words, the deviance is simply the test statistic from the likelihood-ratio test for two nested models, which, as far as I know, has a well-known (asymptotic) distribution. From the Wikipedia:

A convenient result, attributed to Samuel S. Wilks, says that as the sample size $n$ approaches $\infty$, the test statistic $-2 \log(\Lambda)$ for a nested model will be asymptotically $\chi^2$-distributed with degrees of freedom equal to the difference in dimensionality of $\Theta$ and $\Theta_0$.

So, do we know the distribution of $D$, yes or no?

• Only asymptotically and only under the smaller model... – Xi'an Feb 18 '16 at 20:52
• @Xi'an to me that seems like it's almost an answer; certainly it seems to cover the central point. – Glen_b Feb 18 '16 at 22:08

A better idea is that instead of thinking about the sampling distribution you use Posterior samples $\tilde{y}$ using MCMC P($\tilde{y}|y)$ = $\int_\theta P(\tilde{y}|\theta)P(\theta|y)d\theta$ to generate a bunch of Deviance samples $D = -2*\sum_ilog(P(\tilde{y}))$. MCMC methods allow you to generate posterior samples. Since posterior samples encapsulate the uncertainty it is propagated to the deviance and that is what you really want when you make decisions. You use measures like WAIC and LOO which average deviances in very specific ways that are derived from solid theory based in Cross Validation.