Assuming I have two generalised linear models, one with only fixed effects and another with fixed and random effects, how can I compare which model is most parsimonious using AIC/BIC? I can only fit mixed-effect models with glmer, since it returns an error when trying to fit a fixed effect model. I can only fit fixed effect models with glm since this does not allow random effects. This leaves me with two sets of models, one set fitted with glmer and another set fitted with glm.
The accepted answer to the question Can AIC compare across different types of model? states that (with my own emphasis):
It depends. AIC is a function of the log likelihood. If both types of model compute the log likelihood the same way (i.e. include the same constant) then yes you can, if the models are nested.
I'm reasonably certain that glm() and lmer() don't use comparable log likelihoods.
The point about nested models is also up for discussion. Some say AIC is only valid for nested models as that is how the theory is presented/worked through. Others use it for all sorts of comparisons.
The DRAFT r-sig-mixed-models FAQ states that:
Can I use AIC for mixed models? How do I count the number of degrees of freedom for a random effect?
Yes, with caution.
This page then follows by listing a number of caveats and suggestions such as
using modified AIC calculations or REML=FALSE
.
As a novice in this field, I have found that reading about both the use of AIC/BIC for comparing models, and using mixed effect models for group effects to be very eye-opening and also intuitively "common sense" approaches. How to put these two approaches together has left me feeling like I've fallen through the cracks somewhat, there doesn't seem to be a clear answer. Surely if I follow a the AIC/BIC parsimony approach, then I should compare the fixed-effect model with mixed effect model with AIC to see if the random effect deserves to be there?
Am I missing something obvious? How are mixed-effect and fixed-effect generalised linear models usually compared?
glme
? Do you meanglmer
? $\endgroup$