# Odds ratio confidence intervals and p-values suggest different conclusions in a binary logistic mixed-effects model (glmer)

I am running a generalized linear mixed-effects model in R using the glmer function of lme4. The outcome variable is trial-level accuracy in a task (incorrect trials are 0, correct trials are 1), and I have 3, trial-level predictors:

• A, a binary categorical variable
• B, another binary categorical variable
• C, a categorical variable with 3 levels

(All variables, including accuracy are factors.)

Here is the model I'm running:

    acc_model <- glmer(accuracy ~ 1 + A*B*C + (1|subid), data = acc_data, family = binomial,
control = glmerControl(optimizer ="bobyqa"))
#subid simply refers to subject IDs as trials are nested within participants
#I use bobyqa because the model failed to converge with the default optimizer


Then I use the Anova() function from the "car" package to see if the terms are significant.

library(car)
Anova(acc_model)


This has worked nicely for me in the past (anova() does not give p-values for glmer models), and the results seem sensible in this case too:

Analysis of Deviance Table (Type II Wald chisquare tests)

Response: acc
Chisq Df Pr(>Chisq)
A                           10.0468  1   0.001526 **
B                           18.6358  1  1.582e-05 ***
C                            6.7366  2   0.034449 *
A:B                          4.5702  1   0.032532 *
A:C                          0.0142  2   0.992915
B:C                          0.6303  2   0.729685
A:B:C                        2.8599  2   0.239319


The main effects of A, B, C are sig, and so is the A:B two-way interaction.

However, when I calculate odds ratios and corresponding 95% confidence intervals for these effects, the CIs ALL include 1.

This is the code I use to get ORs and CIs:

#CI and OR for variable A
exp(summary(acc_model)$coefficients["A1",1] + qnorm(c(0.025,0.5,0.975)) * summary(acc_model)$coefficients["A1",2])


What can I do here? Is it okay to interpret the Anova() output? I suspect the significance of main effects A, B, and C in the Anova() output could be due to that being a Type II ANOVA (i.e., interactions including a given variable are not controlled for when the main effect of that variable is being evaluated, if I get that correclty), however, the CI for the OR of the A:B interaction ALSO contains 1, and that is NOT involved in any significant higher order interactions.

NOTE: when I use lmerTest, and get summary(acc_model), neither of these variables is significant. I would like to avoid using lmerTest, however, because that does not give an omnibus p-value for categorical variables with more than 2 levels (e.g., Variable C here, that would have 2 separate lines for the Reference_Category - Level1 and the Reference_Category - Level2 contrasts), so anova() and Anova() are neater.

## 1 Answer

The ANOVA is Type II sum of squares and the method you used below is Type III sum of squares. In general, they don't agree. Models for independent data give the same results. For mixed models, they differ. Type II and Type III sum of squares handle the random effect differently. The Type III method assumes, (incorrectly in most cases) that if the "true" data generating fixed effect were different, such as if it were at the lower end of the 95% confidence interval, that the random effect would be unchanged. This is well and good provided the random effects and fixed effects aren't correlated. But they usually are, and the default empirical variance estimates do not suffice to summarize variability.

If you use the confint method in R, there are three confint procedures for mixed models (merMod objects in the nlme package). Wald (Type III, wrong), profile, and bootstrap. Try profile and bootstrap and report back the findings. Note they still may not agree with Type II SS, but the reasons are more refined.

https://www.rdocumentation.org/packages/lme4/versions/1.1-21/topics/confint.merMod

• I don't think sums of squares comes into play with any tests of coefficients in logistic regression, whether they be omnibus tests or tests of single coefficients. – logistic Jul 26 '19 at 21:03
• Thank you, AdamO, this is a very informative response. I had tried both profile and bootstrap CIs but they took an extremely long time to run (I waited around 40 minutes or so for each), so I gave up, and moved on to the method I described above. Do I understand you correctly that that method of calculating CIs is based on Type III sum of squares in some way? – MGy Jul 26 '19 at 23:41
• Sorry, could you provide some further clarification maybe? – MGy Aug 26 '19 at 21:05
• @MGy how many $N$? did you try setting boot replicates very low? – AdamO Aug 26 '19 at 21:18
• N as in the number of observations? I have 34 participants, and around 72 trials per each of them. I let the "boot" version run for a longer time, and it did produce a result albeit with a convergence warning (maxgrad issue), and the resulting CIs still mostly contained 1. – MGy Aug 27 '19 at 22:27