Understanding ANOVA to compare Mixed Model with a GzLM

Question

I'm having a hard time understanding how can I compare a GLM with a GLMM, knowing that I probably can't compare their AIC as glmer from lme4 probably computes the maximum likelihood differently from glm.

However, I wondered if I could find my way using ANOVA that way:

> anova(md.mm, md.logistic1)
Data: d.binary
Models:
md.logistic1: Precision ~ temps + RunProfile
md.mm: Precision ~ temps + RunProfile + (1 | trainId)
             Df    AIC    BIC  logLik deviance  Chisq Chi Df Pr(>Chisq)
md.logistic1  8 1225.6 1276.7 -604.80   1209.6                         
md.mm         9 1225.3 1282.7 -603.63   1207.3 2.3451      1     0.1257

But still, I'm not really sure how to conclude:

1. Is it a proper way to compare those two models ?

2. Can I accurately conclude that considering the degrees of freedom and the not-significant p-value, I should consider the GLM model instead of the GLMM one ?

3. I read that the best way to conclude regarding the significance of random effects in the GLMM model would be to test the null hypothesis about zero variances. I read in the GLMM FAQ that it's possible to test that using the RLRsim package but that only works for lmer models. Is there a way to test it for GLMMs ?

Answer 1

A couple of points:

• Yes, you can compare these two models because model md.logistic1 is nested within model md.mm.
• With the anova() function you do a likelihood ratio test to compare these two models. In particular, the null hypothesis you are testing is that the variance for the random effect you have included for the grouping factor trainId is zero versus the alternative hypothesis that it is not zero.
• A technical/theoretical problem with this likelihood ratio test is that the null hypothesis for this variance parameter is at the boundary of the parameter space. The classic $\chi^2$ distribution that is used to derive the p-value from the test is not valid in this case, namely it will be conservative. To obtain a more correct p-value it has been suggested to assume a mixture of two chi-squared distributions with one having degrees of freedom the number of parameters being tested (in your case 1) and the other having degrees of freedom the number of parameters that are not at the boundary (in your case 0). The RLRsim package performs (an even better) approximation to the p-value. If you are going to use the standard significance level of 5%, you most often have a difference in the conclusion when the original p-value reported by anova() is between 0.05 and 0.10.