2
$\begingroup$

I know that in order to test whether a random effect has a significant impact on a model it's necessary to sequentially remove one random effect at a time and check each model pair with anova() function in lme4 package or through exactLRT() function included in RLRsim package.

However this functions works me well when I worked with lmer() function but not in glmer() function.

In detail, I want to discover if the inclusion of a random effect in my model is significant or not.

model0<-glm(Feed_kg_DM_day~Week, data=dietdef2, family=gaussian(link=log))
model1<-glmer(Feed_kg_DM_day~Week+(1|rat), data=dietdef2, family=gaussian(link=log))

If I perform anova(model0, model1) doesn't show me the p-value:

Analysis of Deviance Table

Model: gaussian, link: log

Response: Feed_kg_DM_day

Terms added sequentially (first to last)


    Df Deviance Resid. Df Resid. Dev
NULL                  2756     1119.1
Week 14   1.5985      2742     1117.5

How can I know that the effect of random variable is significant?

Thanks a lot,

$\endgroup$
  • $\begingroup$ If there are repeated measures / clustering then you shouldnt be testing for significance. Unless the estimated variance is extremely small then retain it without seeking a p value. $\endgroup$ – Robert Long May 26 at 10:33
  • $\begingroup$ And to add to that please read about the many problems with declaring statistical significance, and even more problems when declaring statistical non-significance. $\endgroup$ – Frank Harrell May 26 at 11:09
  • $\begingroup$ @FrankHarrell even though in general I agree with you, how do you propose one should judge whether, e.g., a model with nonlinear effects (e.g., using splines) for BMI, age and LDL cholesterol is/fits better than a model with only the nonlinear effect of age? Aren't a likelihood ratio test between the two models and the corresponding p-value useful in determining which model fits better? $\endgroup$ – Dimitris Rizopoulos May 26 at 18:59
  • $\begingroup$ Details in RMS book and course notes. In short, either user a chunk test to decide to keep all or remove all nonlinear terms, or better pre-specific a model that is as complex as the information content in the data will support, and don't look back. If you think the relationships may not be linear, allow them to be nonlinear. $\endgroup$ – Frank Harrell May 26 at 20:47
3
$\begingroup$

In an experiment with repeated measurements, or an observational study involving clustering, the non-independence of observations within clusters, subjects, participants, is often very well handled by specifying random intercepts for the grouping variable in question.

Unless the estimated variance is very small, the random intercepts should be retained.

Furthermore, testing for significance of the random effects is hindered because we would be trying to formulate a test where, under the null hypothesis, the parameter would be on the boundary of the parameter space (zero), and in any case, removing a patameter from a model in the basis of non-significance is a very questionable thing to do.

$\endgroup$
  • $\begingroup$ But you would still need to determine whether you should include random effects for some of the predictor variables included in the model? Unless you are in a confirmatory setting and are able to fit a maximal model (ncbi.nlm.nih.gov/pmc/articles/PMC3881361)? $\endgroup$ – Isabella Ghement May 26 at 16:00
  • 2
    $\begingroup$ @IsabellaGhement the question is limited to random intercepts, and that was all I wanted to address here. But as to the wider question, Barr et al have a lot to answer for with the terrible general advice to "Keep it Maximal", on the basis of such simple simulations that they employed. See Bates et al (2015l for a rebuttal. $\endgroup$ – Robert Long May 26 at 16:52
  • 1
    $\begingroup$ @IsabellaGhement it's hard to talk to much about this in a comment but my general approach is to include random effects (intercepts or slopes) ONLY when indicated by sound clinical/theoretical justification. $\endgroup$ – Robert Long May 26 at 16:56
  • 1
    $\begingroup$ @RobertLong First of all, I agree with you and Bates et al. (2015) that maximal models are a bad idea. But do you mean that you would not try to include, e.g., non-linear random slopes and see if they improve the model via, e.g., a likelihood ratio test? For example, in longitudinal datasets with 8-10 repeated measurements per subject on average often only random intercepts & random slopes are not enough to capture the correlations in the repeated measurements sufficiently well. $\endgroup$ – Dimitris Rizopoulos May 26 at 18:46
  • $\begingroup$ @DimitrisRizopoulos No I didn't mean to imply that. In the type of situation you describe I would first want to plot visualise the data for each subject, to assess any possible non-linearity before doing what you suggest. $\endgroup$ – Robert Long May 27 at 8:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.