# Comparing non nested models with AIC

Say we have to GLMMs

mod1 <- glmer(y ~ x + A + (1|g), data = dat)
mod2 <- glmer(y ~ x + B + (1|g), data = dat)


These models are not nested in the usual sense of:

a <- glmer(y ~ x + A + (1|g),     data = dat)
b <- glmer(y ~ x + A + B + (1|g), data = dat)


so we can't do anova(mod1, mod2) as we would with anova(a ,b).

Can we use AIC to say which is the best model instead?

## 3 Answers

The AIC can be applied with non nested models. In fact, this is one of the most extended myths (misunderstandings?) about AIC. See:

One thing you have to be careful about is to include all the normalising constants, since these are different for the different (non-nested) models:

See also:

In the context of GLMM a more delicate question is how reliable is the AIC for comparing this sort of models (see also @BenBolker's). Other versions of the AIC are discussed and compared in the following paper:

• note that the marginal vs. conditional AIC distinction is most important when trying to compare models that differ in their sets of random effects – Ben Bolker Sep 27 '14 at 0:55
• @Chandelier & Ben Bolker thank you very much for both your answers. Do either of you happen to have a more formal reference for the argument for using AIC in this way? – user1322296 Sep 27 '14 at 13:53
• @user1322296 I would suggest to go to the root, this is Akaike's paper. AIC is obtained as an estimator of the divergence between your model and the "true model". So, no nesting assumed, only some regularity conditions. – Chandelier Sep 27 '14 at 14:25
• So is it valid to compare the AIC of lm1=x~A+BC and lm2=x ~ D+BC for example? Thanks – crazjo Nov 3 '14 at 7:51
• There appear to be non-nested models for which AIC usage is not appropriate. Here are two examples: 1 and 2. Would you please provide some conditions under which non-nested model selection does work? – Carl Oct 11 '18 at 4:59

For reference, a counterargument: Brian Ripley states in "Selecting amongst large classes of models" pp. 6-7

Crucial assumptions ... The models are nested (footnote: see the bottom of page 615 in the reprint of Akaike (1973)). – AIC is widely used when they are not

The relevant passage (also p. 204 of another reprint of Akaike), starts I think with the phrase "The problem of statistical model identification is often formulated as the problem of selection of $$f(x|_k\theta$$) ...") is not quite available here; I'm looking for a PDF of the paper so I can quote the passage here ...

(I've quoted it below, although honestly at this point I can't see how it supports Ripley's point - it certainly discusses the derivation in the context of nested models but ... ???)

Ripley, B. D. 2004. “Selecting amongst Large Classes of Models.” In Methods and Models in Statistics, edited by N. Adams, M. Crowder, D. J Hand, and D. Stephens, 155–70. London, England: Imperial College Press.

Akaike, H. (1973) Information theory and an extension of the maximum likelihood principle. In Second International Symposium on Information Theory (Eds B. N. Petrov and F. Cáski), pp. 267–281, Budapest. Akademiai Kaidó. Reprinted in Breakthroughs in Statistics , eds Kotz,S. & Johnson, N. L. (1992), volume I, pp. 599–624. New York: Springer. It appears Akaike thought AIC was a useful tool for comparing non-nested models.

"One important observation about AIC is that it is defined without specific reference to the true model [ f(x|kθ) ]. Thus, for any finite number of parametric models, we may always consider an extended model that will play the role of [ f(x|kθ) ] This suggests that AIC can be useful, at least in principle, for the comparison of models which are nonnested, i.e., the situation where the conventional log likelihood-ratio test is not applicable."

(Akaike 1985, pg. 399)

Akaike, Hirotugu. "Prediction and entropy." Selected Papers of Hirotugu Akaike. Springer, New York, NY, 1985. 387-410.