I have a mixed-effects model with two fixed effects and one random effect (group membership) estimated using lme4.

log_dv ~ iv1 + iv2 + (1 | group)

I want to know whether to keep both fixed effects. When I run likelihood ratio tests, the difference between the full and reduced model is significant in both cases, so I keep both fixed effects.

> anova(m.full,m.no_iv1)

refitting model(s) with ML (instead of REML)
Data: month.n.data
Models:
..1: log_dv ~ iv2 + (1 | group)
object: log_dv ~ iv1 + iv2 + (1 | group)
       Df    AIC    BIC  logLik deviance  Chisq Chi Df Pr(>Chisq)    
..1     4 8471.1 8494.9 -4231.6   8463.1                             
object  5 8461.7 8491.5 -4225.9   8451.7 11.374      1  0.0007448 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

> anova(m.no_iv2,m.full)

refitting model(s) with ML (instead of REML)
Data: month.n.data
Models:
object: log_dv ~ iv1 + (1 | group)
..1: log_dv ~ iv1 + iv2 + (1 | group)
       Df    AIC    BIC  logLik deviance  Chisq Chi Df Pr(>Chisq)    
object  4 8774.1 8798.0 -4383.1   8766.1                             
..1     5 8461.7 8491.5 -4225.9   8451.7 314.36      1  < 2.2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

However using the lmerTest package to estimate the degrees of freedom using Kenward-Roger’s approximations, it appears the second fixed effect is not significant.

> anova(m.full, ddf = "Kenward-Roger")
Analysis of Variance Table of type 3  with  Kenward-Roger 
approximation for degrees of freedom
                     Sum Sq Mean Sq NumDF   DenDF F.value    Pr(>F)    
iv1                 11.6921 11.6921     1 167.718 11.4210 0.0009032 ***
iv2                 0.3073  0.3073     1  97.601  0.3001 0.5850409    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

When I look at the confidence intervals for iv2 I am inclined to believe this. Can anyone help me understand what's happening, and suggest what I should do?

  • 2
    it's not the K-R correction (98 df is effectively the same as infinite for these purposes), it has to be something about the way the tests are being defined. What are the results of drop1() ? (The example in ?drop1.merMod shows how to use K-R for this if you want) – Ben Bolker Feb 9 '15 at 15:26
  • 1
    drop1 identifies the error immediately: there are NAs in my data for one of the predictors, so I'm comparing models fit on different data. thank you very much for this and all your other helpful stackoverflow answers! – mloudon Feb 10 '15 at 7:49
  • @BenBolker maybe you could elaborate a little bit more what was your intuition so it could get accepted as an answer (since the comment appeared correct)? – Tim Feb 10 '15 at 9:02
  • @mloudon if possible, could you please give a reproducible example here or send it to the maintainers of the lmerTest package. If there is a potential bug in the lmerTest, we would like then to fix it (but would be nice to have this example to start with) – Alexandra Feb 10 '15 at 9:14
  • Hey Alexandra. This was definitely me and not lmerTest! thanks though – mloudon Feb 14 '15 at 8:58
up vote 2 down vote accepted

For anyone finding this, see comments: the problem was NAs in the predictor variable that was being dropped, causing the model with and without it to be fit on different data

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.