2
$\begingroup$

I have a mixed-effects model with two fixed effects and one random effect (group membership) estimated using lme4. 

log_dv ~ iv1 + iv2 + (1 | group)

I want to know whether to keep both fixed effects. When I run likelihood ratio tests, the difference between the full and reduced model is significant in both cases, so I keep both fixed effects. 

> anova(m.full,m.no_iv1)

refitting model(s) with ML (instead of REML)
Data: month.n.data
Models:
..1: log_dv ~ iv2 + (1 | group)
object: log_dv ~ iv1 + iv2 + (1 | group)
       Df    AIC    BIC  logLik deviance  Chisq Chi Df Pr(>Chisq)    
..1     4 8471.1 8494.9 -4231.6   8463.1                             
object  5 8461.7 8491.5 -4225.9   8451.7 11.374      1  0.0007448 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

> anova(m.no_iv2,m.full)

refitting model(s) with ML (instead of REML)
Data: month.n.data
Models:
object: log_dv ~ iv1 + (1 | group)
..1: log_dv ~ iv1 + iv2 + (1 | group)
       Df    AIC    BIC  logLik deviance  Chisq Chi Df Pr(>Chisq)    
object  4 8774.1 8798.0 -4383.1   8766.1                             
..1     5 8461.7 8491.5 -4225.9   8451.7 314.36      1  < 2.2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

However using the lmerTest package to estimate the degrees of freedom using Kenward-Roger’s approximations, it appears the second fixed effect is not significant. 

> anova(m.full, ddf = "Kenward-Roger")
Analysis of Variance Table of type 3  with  Kenward-Roger 
approximation for degrees of freedom
                     Sum Sq Mean Sq NumDF   DenDF F.value    Pr(>F)    
iv1                 11.6921 11.6921     1 167.718 11.4210 0.0009032 ***
iv2                 0.3073  0.3073     1  97.601  0.3001 0.5850409    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

When I look at the confidence intervals for iv2 I am inclined to believe this. Can anyone help me understand what's happening, and suggest what I should do?

Improve this question
$\endgroup$
5
  • 3
    $\begingroup$ it's not the K-R correction (98 df is effectively the same as infinite for these purposes), it has to be something about the way the tests are being defined. What are the results of drop1() ? (The example in ?drop1.merMod shows how to use K-R for this if you want) $\endgroup$
    – Ben Bolker
    Commented Feb 9, 2015 at 15:26
  • 1
    $\begingroup$ drop1 identifies the error immediately: there are NAs in my data for one of the predictors, so I'm comparing models fit on different data. thank you very much for this and all your other helpful stackoverflow answers! $\endgroup$
    – mloudon
    Commented Feb 10, 2015 at 7:49
  • $\begingroup$ @BenBolker maybe you could elaborate a little bit more what was your intuition so it could get accepted as an answer (since the comment appeared correct)? $\endgroup$
    – Tim
    Commented Feb 10, 2015 at 9:02
  • $\begingroup$ @mloudon if possible, could you please give a reproducible example here or send it to the maintainers of the lmerTest package. If there is a potential bug in the lmerTest, we would like then to fix it (but would be nice to have this example to start with) $\endgroup$
    – Alexandra
    Commented Feb 10, 2015 at 9:14
  • $\begingroup$ Hey Alexandra. This was definitely me and not lmerTest! thanks though $\endgroup$
    – mloudon
    Commented Feb 14, 2015 at 8:58

1 Answer 1

Reset to default
3
$\begingroup$

For anyone finding this, see comments: the problem was NAs in the predictor variable that was being dropped, causing the model with and without it to be fit on different data

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.