Post-hoc tests for unexpected outcome of binomial models

Question

This may be too abstract for SE, in which case I will post in other forums.

I am running a binomial glm, where my response variables should be A and B. Response A is actually a grouping of A1 and A2. I accidentally ran the binomial model with A1, A2, and B as my responses. I understand that if you run a binomial model (in lme4) with more than 2 response types, it will classify the first level as Failure, and all of the other levels as Success. As such, my 2 levels are A1 and (A2, B).

I would expect that A1 and A2 are more similar, and so a model differentiating A vs B would be better fit than a model differentiating A1 vs A2 and B. However, I get more significant predictors when I run the model that groups A2 and B, although the log-likelihood is better for the model differentiating A vs B.

I have reproduced the output below, and am wondering which post-hoc tests I should run to drill down into these results. I also see that the degrees of freedom are different in each model.

Model 1 - A vs B:

Call:
glm(formula = SQ_DA.2 ~ pre_utt_gap + utt_IKI * utt_speed + edit_ct, 
    family = binomial(link = "probit"), data = study1a.SQ.2.df)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-2.6524   0.4943   0.5830   0.6505   1.1996  

Coefficients:
                   Estimate Std. Error z value Pr(>|z|)    
(Intercept)        0.886209   0.036701  24.147  < 2e-16 ***
pre_utt_gap       -0.090382   0.026356  -3.429 0.000605 ***
utt_IKI           -0.077471   0.046980  -1.649 0.099142 .  
utt_speed         -0.159251   0.035318  -4.509 6.51e-06 ***
edit_ct            0.020672   0.005083   4.067 4.77e-05 ***
utt_IKI:utt_speed  0.032871   0.024022   1.368 0.171202    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 3320.2  on 3591  degrees of freedom
Residual deviance: 3260.4  on 3586  degrees of freedom
AIC: 3272.4

> logLik(model1)
'log Lik.' -1630.184 (df=6)

Model 2 - A1 vs A2 and B1:

Call:
glm(formula = SQ_DA.3 ~ pre_utt_gap + utt_IKI * utt_speed + edit_ct, 
    family = binomial(link = "probit"), data = study1a.SQ.2.df)

Deviance Residuals: 
    Min       1Q   Median       3Q      Max  
-1.5925  -1.0244  -0.9236   1.3036   1.8997  

Coefficients:
                   Estimate Std. Error z value Pr(>|z|)    
(Intercept)       -0.176418   0.031157  -5.662 1.49e-08 ***
pre_utt_gap        0.082588   0.022572   3.659 0.000253 ***
utt_IKI            0.098559   0.040861   2.412 0.015863 *  
utt_speed          0.134524   0.030912   4.352 1.35e-05 ***
edit_ct           -0.015926   0.003912  -4.071 4.68e-05 ***
utt_IKI:utt_speed  0.005229   0.021416   0.244 0.807104    
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 4843.1  on 3591  degrees of freedom
Residual deviance: 4792.9  on 3586  degrees of freedom
AIC: 4804.9

> logLik(model2)
'log Lik.' -2396.439 (df=6)

Any suggestions as to where I should look would be greatly appreciated.

Answer 1

The extra "significant" predictor in the second model, utt_IKI, is involved in an interaction with utt_speed. It can be misleading to evaluate the individual regression coefficient for a predictor involved in an interaction.

In the first model, although the interaction isn't "statistically significant" by the p < 0.05 criterion, it's of a reasonable magnitude. It's quite possible that a Wald test combining the two coefficients involving utt_IKI would be significant. Think of the interaction term in the first model as taking some "credit" away from utt_IKI by itself (which is evaluated at the reference/0 level of utt_speed), while in the second model the interaction is negligible so that there is nothing hiding the contribution of utt_IKI to outcome.

If utt_speed is a continuous predictor, you could probably re-center it to make the utt_IKI coefficient pass the (irrelevant, in this case) p < 0.05 criterion. See here as one illustration of what can be going on.

If you think that there is a reason to treat all of A1, A2 and B separately, you could do a multinomial or ordinal regression with all three as possible outcomes, depending on whether those categories have an inherent order to them.