# Likelihood ratio test - lmer R - Non-nested models

I am currently reviewing some work and have come across the following, which seems wrong to me. Two mixed models are fitted (in R) using lmer. The models are non-nested and are compared by likelihood-ratio tests. In short, here is a reproducible example of what I have:

set.seed(105)
Resp = rnorm(100)
A = factor(rep(1:5,each=20))
B = factor(rep(1:2,times=50))
C = rep(1:4, times=25)
m1 = lmer(Resp ~ A + (1|C), REML = TRUE)
m2 = lmer(Resp ~ B + (1|C), REML = TRUE)
anova(m1,m2)


As far as I can see, lmer is used to compute the log-likelihood and the anova statement tests the difference between the models using a chi-square with the usual degrees of freedom. This does not seem correct to me. If it is correct, does anyone know of any reference justifying this? I am aware of methods relying on simulations (Paper by Lewis et al., 2011) and the approach developed by Vuong (1989) but I do not think that this is what is produced here. I do not think that the use of the anova statement is correct.

This is not correct in two ways:

1. (Ordinary) likelihood ratio test can only be used to compare nested models;
2. We cannot compare mean models under REML. (This is not the case here, see @KarlOveHufthammer's comments below.)

In the case of using ML, I am aware of using AIC or BIC to compare the non-nested models.

• Regarding point 2, the anova() function in R does not compare the two models fitted under REML; it refits them using ML and then perform the test. See lme4:::anova.merMod, which contains the line mods <- lapply(mods, refitML). (But you are still right that anova() can’t be used to compare the two models, as they are not nested.) Commented Jan 29, 2014 at 19:32
• also note that there is some disagreement on nesting: Brian Ripley says nesting is essential for AIC comparison (see p. 20 of linked document for discussion), while Anderson and Burnham (see p. 2) disagree .. Commented Jan 29, 2014 at 23:43
• @BenBolker Another reference (see also this and this) for the use of AIC with non nested models, as long as you consider all the normalising constants as well as non-pathological models. In the context of LMM, however, you have to use some modifications of the AIC. Commented Jan 29, 2014 at 23:52
• Link mangled: I think stats.ox.ac.uk/~ripley/ModelChoice.pdf should work. Commented Jan 30, 2014 at 20:38
• @BenBolker Well, Brian Ripley is quite opinionated. However, he hasn't provided a devastating argument against the use of AIC for non nested models :). Sorry for repeating your link. Commented Jan 30, 2014 at 23:46