What to conclude about these models? Random intercept + Fixed Slope vs. Random intercept and Slope

Question

In the 'nlme' R package, for instance, I ran the following models:

model1 <- lme(y~x, random=~x|group, data=data)
model2 <- lme(y~x, random=~1|group, data=data)

And then I compared them with an ANOVA:

anova(model1, model2)

       Model df       AIC       BIC   logLik   Test  L.Ratio p-value
model1     1  6 -310.3753 -289.7468 161.1876                        
model2     2  4 -307.4426 -293.6903 157.7213 1 vs 2 6.932679  0.0312

I am confused about how to interpret that output. Correct me if I am wrong, but in model1 I allow my intercept and slope to be random, as opposed to model2, where only my intercept has a random effect. Model1 AIC is better than model2, but model2 BIC is better than model1. However, the anova table tells me that the models are different from each other. With that in mind, I wonder which one is the best model and how could I justify that?

Answer 1

"Best" for what?

• AIC attempts to identify the model that will best predict future data (it minimizes the expected Kullback-Leibler distance, which is a measure of the distance between predicted and actual values of outcomes). AIC is asymptotically equivalent to cross-validation; well-implemented cross-validation [i.e., resampling at the level of groups] will make fewer approximations, and hence will be more reliable although more computationally expensive.
• BIC attempts to identify the true model (it is an asymptotic approximation of the Bayes factor, which is related to the Bayesian posterior probability of the model)
• the log-likelihood ratio test attempts to reject the null hypothesis (that the slopes are identical for every group)

Since these are all different questions, it's confusing but not horribly surprising when they give different answers, although in extreme cases (the models are very nearly identical, or one model is much better than the other) they generally agree. I would use whichever criterion is more consistent with the general statistical approach you're taking (which ideally you decided on before you started to analyze the data ...)

In general AIC is less conservative than BIC (provided you have more than 8 data points), so it's not surprising that AIC chooses the more complex models.

You should also be aware that all of these tests are moderately crude approximations in the mixed model case:

• a null hypothesis on the border of the feasible space violates the assumptions of the derivation of the AIC and LRT, making them generally conservative
• counting number of observations for the BIC is tricky: do we count groups or observations?
• for the AIC, are we trying to optimize prediction at the population or the individual level ("level of focus")?

These issues are discussed somewhat further, with references, in the GLMM FAQ, specifically the sections on testing significance of random effects and Can I use AIC for mixed models?