# Post-hoc tests for unexpected outcome of binomial models

This may be too abstract for SE, in which case I will post in other forums.

I am running a binomial glm, where my response variables should be A and B. Response A is actually a grouping of A1 and A2. I accidentally ran the binomial model with A1, A2, and B as my responses. I understand that if you run a binomial model (in lme4) with more than 2 response types, it will classify the first level as Failure, and all of the other levels as Success. As such, my 2 levels are A1 and (A2, B).

I would expect that A1 and A2 are more similar, and so a model differentiating A vs B would be better fit than a model differentiating A1 vs A2 and B. However, I get more significant predictors when I run the model that groups A2 and B, although the log-likelihood is better for the model differentiating A vs B.

I have reproduced the output below, and am wondering which post-hoc tests I should run to drill down into these results. I also see that the degrees of freedom are different in each model.

Model 1 - A vs B:

Call:
glm(formula = SQ_DA.2 ~ pre_utt_gap + utt_IKI * utt_speed + edit_ct,
family = binomial(link = "probit"), data = study1a.SQ.2.df)

Deviance Residuals:
Min       1Q   Median       3Q      Max
-2.6524   0.4943   0.5830   0.6505   1.1996

Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept)        0.886209   0.036701  24.147  < 2e-16 ***
pre_utt_gap       -0.090382   0.026356  -3.429 0.000605 ***
utt_IKI           -0.077471   0.046980  -1.649 0.099142 .
utt_speed         -0.159251   0.035318  -4.509 6.51e-06 ***
edit_ct            0.020672   0.005083   4.067 4.77e-05 ***
utt_IKI:utt_speed  0.032871   0.024022   1.368 0.171202
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

Null deviance: 3320.2  on 3591  degrees of freedom
Residual deviance: 3260.4  on 3586  degrees of freedom
AIC: 3272.4

> logLik(model1)
'log Lik.' -1630.184 (df=6)


Model 2 - A1 vs A2 and B1:

Call:
glm(formula = SQ_DA.3 ~ pre_utt_gap + utt_IKI * utt_speed + edit_ct,
family = binomial(link = "probit"), data = study1a.SQ.2.df)

Deviance Residuals:
Min       1Q   Median       3Q      Max
-1.5925  -1.0244  -0.9236   1.3036   1.8997

Coefficients:
Estimate Std. Error z value Pr(>|z|)
(Intercept)       -0.176418   0.031157  -5.662 1.49e-08 ***
pre_utt_gap        0.082588   0.022572   3.659 0.000253 ***
utt_IKI            0.098559   0.040861   2.412 0.015863 *
utt_speed          0.134524   0.030912   4.352 1.35e-05 ***
edit_ct           -0.015926   0.003912  -4.071 4.68e-05 ***
utt_IKI:utt_speed  0.005229   0.021416   0.244 0.807104
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

Null deviance: 4843.1  on 3591  degrees of freedom
Residual deviance: 4792.9  on 3586  degrees of freedom
AIC: 4804.9

> logLik(model2)
'log Lik.' -2396.439 (df=6)


Any suggestions as to where I should look would be greatly appreciated.

The extra "significant" predictor in the second model, utt_IKI, is involved in an interaction with utt_speed. It can be misleading to evaluate the individual regression coefficient for a predictor involved in an interaction.

In the first model, although the interaction isn't "statistically significant" by the p < 0.05 criterion, it's of a reasonable magnitude. It's quite possible that a Wald test combining the two coefficients involving utt_IKI would be significant. Think of the interaction term in the first model as taking some "credit" away from utt_IKI by itself (which is evaluated at the reference/0 level of utt_speed), while in the second model the interaction is negligible so that there is nothing hiding the contribution of utt_IKI to outcome.

If utt_speed is a continuous predictor, you could probably re-center it to make the utt_IKI coefficient pass the (irrelevant, in this case) p < 0.05 criterion. See here as one illustration of what can be going on.

If you think that there is a reason to treat all of A1, A2 and B separately, you could do a multinomial or ordinal regression with all three as possible outcomes, depending on whether those categories have an inherent order to them.

• Thanks! This is extremely helpful. If I do run a multinomial model with the 3 responses, what's the best way to compare it to the binomial model? Should I just use R^2 or some other goodness of fit? Commented Jun 14, 2022 at 19:52
• @Adam_G R^2 isn't so helpful for examining goodness-of-fit of a binomial or multinomial model. At least that's true of the usual R^2 measures; there are some types of "pseudo-R^2" for models fit by maximum likelihood. Do your three outcome categories have a natural ordering, like A1 < A2 < B in some sense? Then you should use ordinal regression. Also, ask yourself what specific hypothesis you want to test: what does it mean to compare a model predicting 3 outcomes to one predicting only 2?
– EdM
Commented Jun 14, 2022 at 20:21
• I'm comparing types of sentences, and how people type them (all the predictors are typing patterns/timings). In this case A1=Statement-non-opinion, A2=Statement-opinion, and A3=Question. A vs B is testing Statement vs Question. So in answer to your question, there is no inherent order. Commented Jun 14, 2022 at 20:53
• @Adam_G then think about which really is the "predictor" here. If people are given different types of sentences to type, then type of sentence would seem to be the independent/predictor variable, while "how people type them" would seem to be the set of dependent/outcome variables. Then you could do a true "multivariate" analysis of variance that takes into account correlations among the outcomes. Also, if the same individual is given multiple sentences to type, you need to take that correlation within individuals into account in the model, for example with a random-effects model.
– EdM
Commented Jun 14, 2022 at 21:02
• My experiment was actually a spontaneous conversation; no one was given anything to type. In this case my question is "Are there significant differences in the way people type statements vs questions?" The 2nd half of my study will actually be typing-pattern ~ sentence-type, to answer the question of whether sentence types have distinct typing signatures, even if some of those signatures are the same for some sentence types (there are 10 different types). Commented Jun 14, 2022 at 21:07