# What are easy to interpret, goodness of fit measures for linear mixed effects models?

I am currently using the R package lme4.

I am using a linear mixed effects models with random effects:

library(lme4)
mod1 <- lmer(r1 ~ (1 | site), data = sample_set) #Only random effects
mod2 <- lmer(r1 ~ p1 + (1 | site), data = sample_set) #One fixed effect +
# random effects
mod3 <- lmer(r1 ~ p1 + p2 + (1 | site), data = sample_set) #Two fixed effects +
# random effects


To compare models, I am using the anova function and looking at differences in AIC relative to the lowest AIC model:

anova(mod1, mod2, mod3)


The above is fine for comparing models.

However, I also need some simple way to interpret goodness of fit measures for each model. Does anyone have experience with such measures? I have done some research, and there are journal papers on R squared for the fixed effects of mixed effects models:

• Cheng, J., Edwards, L. J., Maldonado-Molina, M. M., Komro, K. A., & Muller, K. E. (2010). Real longitudinal data analysis for real people: Building a good enough mixed model. Statistics in Medicine, 29(4), 504-520. doi: 10.1002/sim.3775
• Edwards, L. J., Muller, K. E., Wolfinger, R. D., Qaqish, B. F., & Schabenberger, O. (2008). An R2 statistic for fixed effects in the linear mixed model. Statistics in Medicine, 27(29), 6137-6157. doi: 10.1002/sim.3429

It seems however, that there is some criticism surrounding the use of measures such as those proposed in the above papers.

Could someone please suggest a few easy to interpret, goodness of fit measures that could apply to my models?

-
I really like the question, but using likelihood ratio tests to determine whether or not fixed effects are needed is not the recommended strategy, see the faq. So the above is not fine for comparing models. –  Henrik Sep 25 '12 at 16:39
Thanks Henrik. The FAQ you listed is very helpful. It sounds like Markov chain Monte Carlo sampling could be a good strategy to compare my models. –  mjburns Sep 26 '12 at 0:07
The problem with MCMC is that you can only have simple random effects (as in your example). I would go with kenward-rogers approximation to degrees of freedom as it also applies to more complicated models. Have a look at function mixed() in my afex package (the development version also has parametric bootstrap). See here for some references. –  Henrik Sep 26 '12 at 10:21
OK Henrik. I managed to get your mixed() function working from the afex package. Could you please advise on how I could use afex to compare models? What measure(s) could I use to decide if one model is more plausible than another? Thanks. –  mjburns Sep 26 '12 at 11:34
This is not easily answered, perhaps you ask a separate question giving more details. But just briefly, afex tries to help you to assess whether certain effects (or better models including this effect) are significant. To this end it uses KRmodcomp from package pbkrtest. You can also use KRmodcompdirectly to compare models. –  Henrik Sep 27 '12 at 13:48
show 1 more comment

Random effect fit (mod1) can be measured by ICC and ICC2 (the ratio between variance accounted by random effects and the residual variance). psychometric R package includes a function to extract them form a lme object.
It is possible to use R2 to assess fixed effect (mod2, mod3), but this can be tricky: When two models show a similar R2 it can be the case that one is more "accurate", but that is masked by its fixed factor "subtracting" a greater variance component to the random effect. On the other hand it is easy to interpret a greater R2 of the highest order model (eg mod3). In Baayen's chapter on mixed models there is a nice discussion about this. Also, it's tutorial is very clear.
A possible solution is to consider each variance component independently, and then use them to compare the models.