Odds ratio confidence intervals and p-values suggest different conclusions in a binary logistic mixed-effects model (glmer)

Question

I am running a generalized linear mixed-effects model in R using the glmer function of lme4. The outcome variable is trial-level accuracy in a task (incorrect trials are 0, correct trials are 1), and I have 3, trial-level predictors:

• A, a binary categorical variable
• B, another binary categorical variable
• C, a categorical variable with 3 levels

(All variables, including accuracy are factors.)

Here is the model I'm running:

    acc_model <- glmer(accuracy ~ 1 + A*B*C + (1|subid), data = acc_data, family = binomial,
                   control = glmerControl(optimizer ="bobyqa"))
    #subid simply refers to subject IDs as trials are nested within participants
    #I use bobyqa because the model failed to converge with the default optimizer

Then I use the Anova() function from the "car" package to see if the terms are significant.

library(car)
Anova(acc_model)

This has worked nicely for me in the past (anova() does not give p-values for glmer models), and the results seem sensible in this case too:

Analysis of Deviance Table (Type II Wald chisquare tests)

Response: acc
                              Chisq Df Pr(>Chisq)    
A                           10.0468  1   0.001526 ** 
B                           18.6358  1  1.582e-05 ***
C                            6.7366  2   0.034449 *  
A:B                          4.5702  1   0.032532 *  
A:C                          0.0142  2   0.992915    
B:C                          0.6303  2   0.729685    
A:B:C                        2.8599  2   0.239319    

The main effects of A, B, C are sig, and so is the A:B two-way interaction.

However, when I calculate odds ratios and corresponding 95% confidence intervals for these effects, the CIs ALL include 1.

This is the code I use to get ORs and CIs:

#CI and OR for variable A
exp(summary(acc_model)$coefficients["A1",1] + qnorm(c(0.025,0.5,0.975)) * summary(acc_model)$coefficients["A1",2])

What can I do here? Is it okay to interpret the Anova() output? I suspect the significance of main effects A, B, and C in the Anova() output could be due to that being a Type II ANOVA (i.e., interactions including a given variable are not controlled for when the main effect of that variable is being evaluated, if I get that correclty), however, the CI for the OR of the A:B interaction ALSO contains 1, and that is NOT involved in any significant higher order interactions.

NOTE: when I use lmerTest, and get summary(acc_model), neither of these variables is significant. I would like to avoid using lmerTest, however, because that does not give an omnibus p-value for categorical variables with more than 2 levels (e.g., Variable C here, that would have 2 separate lines for the Reference_Category - Level1 and the Reference_Category - Level2 contrasts), so anova() and Anova() are neater.

Answer 1

The ANOVA is Type II sum of squares and the method you used below is Type III sum of squares. In general, they don't agree. Models for independent data give the same results. For mixed models, they differ. Type II and Type III sum of squares handle the random effect differently. The Type III method assumes, (incorrectly in most cases) that if the "true" data generating fixed effect were different, such as if it were at the lower end of the 95% confidence interval, that the random effect would be unchanged. This is well and good provided the random effects and fixed effects aren't correlated. But they usually are, and the default empirical variance estimates do not suffice to summarize variability.

If you use the confint method in R, there are three confint procedures for mixed models (merMod objects in the nlme package). Wald (Type III, wrong), profile, and bootstrap. Try profile and bootstrap and report back the findings. Note they still may not agree with Type II SS, but the reasons are more refined.

https://www.rdocumentation.org/packages/lme4/versions/1.1-21/topics/confint.merMod