# Comparing non nested models with AIC

Say we have to GLMMs

mod1 <- glmer(y ~ x + A + (1|g), data = dat)
mod2 <- glmer(y ~ x + B + (1|g), data = dat)


These models are not nested in the usual sense of:

a <- glmer(y ~ x + A + (1|g),     data = dat)
b <- glmer(y ~ x + A + B + (1|g), data = dat)


so we can't do anova(mod1, mod2) as we would with anova(a ,b).

Can we use AIC to say which is the best model instead?

The AIC can be applied with non nested models. In fact, this is one of the most extended myths (misunderstandings?) about AIC. See:

One thing you have to be careful about is to include all the normalising constants, since these are different for the different (non-nested) models:

In the context of GLMM a more delicate question is how reliable is the AIC for comparing this sort of models (see also @BenBolker's). Other versions of the AIC are discussed and compared in the following paper:

• note that the marginal vs. conditional AIC distinction is most important when trying to compare models that differ in their sets of random effects Sep 27, 2014 at 0:55
• @Chandelier & Ben Bolker thank you very much for both your answers. Do either of you happen to have a more formal reference for the argument for using AIC in this way? Sep 27, 2014 at 13:53
• @user1322296 I would suggest to go to the root, this is Akaike's paper. AIC is obtained as an estimator of the divergence between your model and the "true model". So, no nesting assumed, only some regularity conditions. Sep 27, 2014 at 14:25
• So is it valid to compare the AIC of lm1=x~A+BC and lm2=x ~ D+BC for example? Thanks Nov 3, 2014 at 7:51
• There appear to be non-nested models for which AIC usage is not appropriate. Here are two examples: 1 and 2. Would you please provide some conditions under which non-nested model selection does work?
– Carl
Oct 11, 2018 at 4:59

For reference, a counterargument: Brian Ripley states in "Selecting amongst large classes of models" pp. 6-7

Crucial assumptions ... The models are nested (footnote: see the bottom of page 615 in the reprint of Akaike (1973)). – AIC is widely used when they are not

The relevant passage (also p. 204 of another reprint of Akaike), starts I think with the phrase "The problem of statistical model identification is often formulated as the problem of selection of $$f(x|_k\theta$$) ...") is not quite available here; I'm looking for a PDF of the paper so I can quote the passage here ...

(I've quoted it below, although honestly at this point I can't see how it supports Ripley's point - it certainly discusses the derivation in the context of nested models but ... ???)

Ripley, B. D. 2004. “Selecting amongst Large Classes of Models.” In Methods and Models in Statistics, edited by N. Adams, M. Crowder, D. J Hand, and D. Stephens, 155–70. London, England: Imperial College Press.

Akaike, H. (1973) Information theory and an extension of the maximum likelihood principle. In Second International Symposium on Information Theory (Eds B. N. Petrov and F. Cáski), pp. 267–281, Budapest. Akademiai Kaidó. Reprinted in Breakthroughs in Statistics , eds Kotz,S. & Johnson, N. L. (1992), volume I, pp. 599–624. New York: Springer. 