AIC and anova p in multilevel model, how to interpret?

Question

I have a model with a random-nested factor, I am comparing it with a model without the random factor (to test significance of random factor) as follows:

M2<-lme(score ~ disease* time* week, random=~1|treatment/Id, method = "REML", 
        data = Dat1)
M3<-gls(score ~ disease * time * week, method = "REML", data = Dat1)

Comparing the two I get:

df         AIC
M2  21 -2662.715    
M3  19 -2612.308

This leads me to believe M3 would be a better model (AIC closer to 0), meaning random could be taken out, or at least that M2 and M3 are different.

I check with ANOVA:

  Model   df    AIC        BIC      logLik     Test    L.Ratio    p-value
M2     1  21  -2662.715  -2550.233  1352.358                        
M3     2  19  -2612.308  -2510.538  1325.154   1 vs 2  54.40752   <.0001

and p is significant p<.0001.

I am not sure how to interpret this, M2 and M3 are significantly different, but does this mean random effect is a significant factor, or does it mean M3 is a better model and the therefore random effect is not significant?

Answer 1

The model with the lower AIC is the "better" model. In your example, model2 is the better model, and the random factor is significant.

The significant p-value means that model2 and model3 are significantly different, and if model2 contains more information than model3, that means model3 is not adequate compared to model2.

Edit: However, since your models were fitted by REML, the reported likelihood ratio test p-value is off (not enough to change the test result in your case but may for other cases); the critical value that you should compare the likelihood ratio test statistic against is 5.14 (significance of 0.05) from a mixture of chi-squared distributions.