2
$\begingroup$

When models fitted using REML are compared, the fixed structure needs to be the same between all models. However, when models are fit using ML (to compare fixed effects), does the random structure need to be the same?

from Zuur et al. (2009; PAGE 122):

To compare models with nested fixed effects (but with the same random structure), ML estimation must be used and not REML.

This suggests to me that yes, the random effects need to be the same. [Zuur et al. 2009. Mixed Effect Models and Extensions in Ecology with R. Springer.]

however, from Doug Bates SASmixed vignette:

When models are fit by maximum likelihood, ...these quality-of-fit criteria can be used to evaluate different fixed-effects specifications or different random-effects specifications or different specifications of both fixed effects and random effects

Am I misinterpreting the Zuur et al (2009) quote?

$\endgroup$
1
$\begingroup$

Zuur is describing but one setting for testing mixed models. For the case that models have nested fixed effects and identical random effect structure(s), the likelihood ratio statistic can be compared to a $\chi^2_{d}$ distribution ($d$ the difference in the number of parameters of the "big" model and of the "smaller" model). Zuur did not say there are no tests for the class of models Bates describes.

Bates is right that you can test for one or more random effects. The issue is that they aren't the type of tests you're used to. Say you're just testing the presence of a random intercept; the distribution of the likelihood ratio statistic is not even asymptotically $\chi^2_1$, but rather it is a mixture of $\chi^2_1$ random variables. This has nothing to do with whether the model is fit with REML or ML.

As Bates suggests, the actual distribution may be moot if one is merely interested in predictive accuracy and trying to rank models (using LRT, AIC, or BIC). He is careful not to use the word "test" but rather evaluate. However, the ICs will penalize the parameters of the model. In general, the likelihood will always improve by adding parameters to a model, as is true in the fixed case, so I might disagree the LR-statistic is any good for comparing models without knowing its asymptotic distribution.

If you're interested in learning more about tests of variance components, there's a lot of literature and it's still an open research area in many ways. A nice overview is given here:

https://www4.stat.ncsu.edu/~dzhang2/st755/vartest.pdf

$\endgroup$
  • $\begingroup$ So if I were trying to perform model selection for predictive purposes and had 2-3 different possible random effect structures and a handful of fixed effects, it would be ok to use information criteria to perform model selection from whatever mixed bag of combinations that may be of interest? $\endgroup$ – user2602640 Mar 19 '18 at 21:01
  • $\begingroup$ @user2602640 I guess. I don't think fixed/random effects should be penalized the same in AIC or BIC. Still, I've never found a use for random effects for prediction. How do you ever observe the random term? I've always bucked the idea in favor of fixed effects all the way. $\endgroup$ – AdamO Mar 19 '18 at 21:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.