Allowed comparisons of mixed effects models (random effects primarily)

Question

I've been looking at mixed effects modelling using the lme4 package in R. I'm primarily using the lmer command so I'll pose my question through code that uses that syntax. I suppose a general easy question might be, is it OK to compare any two models constructed in lmer using likelihood ratios based on identical datasets? I believe the answer to that must be, "no", but I could be incorrect. I've read conflicting information on whether the random effects have to be the same or not, and what component of the random effects is meant by that? So, I'll present a few examples. I'll take them from repeated measures data using word stimuli, perhaps something like Baayen (2008) would be useful in interpreting.

Let's say I have a model where there are two fixed effects predictors, we'll call them A, and B, and some random effects... words and subjects that perceived them. I might construct a model like the following.

m <- lmer( y ~ A + B + (1|words) + (1|subjects) )

(note that I've intentionally left out data = and we'll assume I always mean REML = FALSE for clarity's sake)

Now, of the following models, which are OK to compare with a likelihood ratio to the one above and which are not?

m1 <- lmer( y ~ A + B + (A+B|words) + (1|subjects) )
m2 <- lmer( y ~ A + B + (1|subjects) )              
m3 <- lmer( y ~ A + B + (C|words) + (A+B|subjects) )
m4 <- lmer( y ~ A + B + (1|words) )                 
m5 <- lmer( y ~ A * B + (1|subjects) )   

I acknowledge that the interpretation of some of these differences may be difficult, or impossible. But let's put that aside for a second. I just want to know if there's something fundamental in the changes here that precludes the possibility of comparing. I also want to know whether, if LRs are OK, and AIC comparisons as well.

Answer 1

Using maximum likelihood, any of these can be compared with AIC; if the fixed effects are the same (m1 to m4), using either REML or ML is fine, with REML usually preferred, but if they are different, only ML can be used. However, interpretation is usually difficult when both fixed effects and random effects are changing, so in practice, most recommend changing only one or the other at a time.

Using the likelihood ratio test is possible but messy because the usual chi-squared approximation doesn't hold when testing if a variance component is zero. See Aniko's answer for details. (Kudos to Aniko for both reading the question more carefully than I did and reading my original answer carefully enough to notice that it missed this point. Thanks!)

Pinhiero/Bates is the classic reference; it describes the nlme package, but the theory is the same. Well, mostly the same; Doug Bates has changed his recommendations on inference since writing that book and the new recommendations are reflected in the lme4 package. But that's more than I want to get into here. A more readable reference is Weiss (2005), Modeling Longitudinal Data.

Answer 2

You have to be careful using likelihood-ratio tests when evaluating whether a variance component is 0 (m vs m-m4), because the typical chi-square approximation does not apply. The reason is that the null-hypothesis is $\sigma^2=0$, and it is on the boundary of the parameter space, so the classical results do not apply.

There is an entire theory of the distribution of LRT in these situations (see, for example, Self and Liang, 1987 [1]), however it quickly becomes messy. For the special case of only one parameter hitting the boundary (eg, m vs m2), the likelihood ratio has a $\frac 12 \chi^2_1 + \frac 12 \chi^2_0$ distribution, which in practice means that you have to halve your p-value based on $\chi^2_1$.

However, as @Aaron stated, many experts do not recommend doing a likelihood ratio test like this. Potential alternatives are the information criteria (AIC, BIC, etc), or bootstrapping the LRT.

[1] Self, S. G. & Liang, K. Asymptotic properties of maximum likelihood estimators and likelihood ratio tests under nonstandard conditions J. Amer. Statist. Assoc., 1987, 82, 605-610.