Comparing non nested models with AIC

Question

Say we have to GLMMs

mod1 <- glmer(y ~ x + A + (1|g), data = dat)
mod2 <- glmer(y ~ x + B + (1|g), data = dat)

These models are not nested in the usual sense of:

a <- glmer(y ~ x + A + (1|g),     data = dat)
b <- glmer(y ~ x + A + B + (1|g), data = dat)

so we can't do anova(mod1, mod2) as we would with anova(a ,b).

Can we use AIC to say which is the best model instead?

Answer 1

The AIC can be applied with non nested models. In fact, this is one of the most extended myths (misunderstandings?) about AIC. See:

• Akaike Information Criterion

• AIC MYTHS AND MISUNDERSTANDINGS

One thing you have to be careful about is to include all the normalising constants, since these are different for the different (non-nested) models:

See also:

• Non-nested model selection

• AIC for non-nested models: normalizing constant

In the context of GLMM a more delicate question is how reliable is the AIC for comparing this sort of models (see also @BenBolker's). Other versions of the AIC are discussed and compared in the following paper:

• On the behaviour of marginal and conditional AIC in linear mixed models

Answer 2

For reference, a counterargument: Brian Ripley states in "Selecting amongst large classes of models" pp. 6-7

Crucial assumptions ... The models are nested (footnote: see the bottom of page 615 in the reprint of Akaike (1973)). – AIC is widely used when they are not

The relevant passage (also p. 204 of another reprint of Akaike), starts I think with the phrase "The problem of statistical model identification is often formulated as the problem of selection of $f(x|_k\theta$) ...") is not quite available here; I'm looking for a PDF of the paper so I can quote the passage here ...

(I've quoted it below, although honestly at this point I can't see how it supports Ripley's point - it certainly discusses the derivation in the context of nested models but ... ???)

Ripley, B. D. 2004. “Selecting amongst Large Classes of Models.” In Methods and Models in Statistics, edited by N. Adams, M. Crowder, D. J Hand, and D. Stephens, 155–70. London, England: Imperial College Press.

Akaike, H. (1973) Information theory and an extension of the maximum likelihood principle. In Second International Symposium on Information Theory (Eds B. N. Petrov and F. Cáski), pp. 267–281, Budapest. Akademiai Kaidó. Reprinted in Breakthroughs in Statistics , eds Kotz,S. & Johnson, N. L. (1992), volume I, pp. 599–624. New York: Springer.

For reference, see also this post on MathOverflow.

---

Answer 3

It appears Akaike thought AIC was a useful tool for comparing non-nested models.

"One important observation about AIC is that it is defined without specific reference to the true model [ f(x|kθ) ]. Thus, for any finite number of parametric models, we may always consider an extended model that will play the role of [ f(x|kθ) ] This suggests that AIC can be useful, at least in principle, for the comparison of models which are nonnested, i.e., the situation where the conventional log likelihood-ratio test is not applicable."

(Akaike 1985, pg. 399)

Akaike, Hirotugu. "Prediction and entropy." Selected Papers of Hirotugu Akaike. Springer, New York, NY, 1985. 387-410.