I'm having a hard time understanding how can I compare a GLM with a GLMM, knowing that I probably can't compare their AIC as glmer from lme4 probably computes the maximum likelihood differently from glm.

However, I wondered if I could find my way using ANOVA that way:

> anova(md.mm, md.logistic1)
Data: d.binary
md.logistic1: Precision ~ temps + RunProfile
md.mm: Precision ~ temps + RunProfile + (1 | trainId)
             Df    AIC    BIC  logLik deviance  Chisq Chi Df Pr(>Chisq)
md.logistic1  8 1225.6 1276.7 -604.80   1209.6                         
md.mm         9 1225.3 1282.7 -603.63   1207.3 2.3451      1     0.1257

But still, I'm not really sure how to conclude:

  1. Is it a proper way to compare those two models ?

  2. Can I accurately conclude that considering the degrees of freedom and the not-significant p-value, I should consider the GLM model instead of the GLMM one ?

  3. I read that the best way to conclude regarding the significance of random effects in the GLMM model would be to test the null hypothesis about zero variances. I read in the GLMM FAQ that it's possible to test that using the RLRsim package but that only works for lmer models. Is there a way to test it for GLMMs ?


A couple of points:

  • Yes, you can compare these two models because model md.logistic1 is nested within model md.mm.
  • With the anova() function you do a likelihood ratio test to compare these two models. In particular, the null hypothesis you are testing is that the variance for the random effect you have included for the grouping factor trainId is zero versus the alternative hypothesis that it is not zero.
  • A technical/theoretical problem with this likelihood ratio test is that the null hypothesis for this variance parameter is at the boundary of the parameter space. The classic $\chi^2$ distribution that is used to derive the p-value from the test is not valid in this case, namely it will be conservative. To obtain a more correct p-value it has been suggested to assume a mixture of two chi-squared distributions with one having degrees of freedom the number of parameters being tested (in your case 1) and the other having degrees of freedom the number of parameters that are not at the boundary (in your case 0). The RLRsim package performs (an even better) approximation to the p-value. If you are going to use the standard significance level of 5%, you most often have a difference in the conclusion when the original p-value reported by anova() is between 0.05 and 0.10.
  • $\begingroup$ Thanks for your feedback. Regarding the last point you mention, I read many times that the Chi-square p-value result had to be divided by 2 in order to be considered accurate. Is it a good general rule of thumb or am I mistaken ? From GLMM-FAQ: "in the simplest case (single random effect variance), the p-value is approximately twice as large as it should be (Pinheiro and Bates 2000)". $\endgroup$
    – accpnt
    Jul 22 '19 at 12:25
  • $\begingroup$ This is only in your case where you compare the model without random effects with a model with random intercepts alone - not in general. $\endgroup$ Jul 22 '19 at 13:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.