Question about post hoc analyses for mixed-effects logistic regression model

Question

I'm sorry for such a silly question, but I would like to ask about post hoc comparisons that I could use to follow up on a significant, although hard to interpret, three-way interaction.

I ran a mixed-effects logistic regression model with glmer() predicting categorization responses from the speech cue weighting task with the following formula:

response ~ Time * Group * pitch_level * duration_level

where Time has 3 levels (Time I, Time II, Time III) coded with contr.sdif(3) to compare subsequent differences between Time I vs Time II, and then Time II vs Time III, Group has 2 levels (experimental or control), and pitch_level and duration_level both have 4 levels of how much of pitch or duration information was there in a given trial.

I have a significant 3-way interaction of Time (I-II) * Group * pitch_level that suggests some differences from Time I to II between the groups in how much they rely on pitch while categorizing speech, but to understand which group uses more pitch than the other and at which testing time, we need to run some post hoc analyses. Is that correct?

What post hoc analyses do you think would be appropriate here?

I can think of two options: (1) One is to break down the model by Group to see if the significant effect of time is still there - but this would be effectively another analysis which does not take into account all the dataset but only its subsets. Isn't problematic? How would I interpret significant two-way interactions of Time x pitch_level in each group then? (2) Second, to compare performance across Time x Group x pitch levels with emmeans() across all the pitch levels, but then we do not take into account the interaction effects since these are just pairwise comparisons.

Do you think any of these make sense? Or is there anything obvious I'm missing here?

I would very much appreciate your advice, thanks!

Answer 1

Welcome to CV!

You say you fitted a mixed-effects model, but I don't see any random factors in the model formula. Am I missing something? If not, then it's really a fixed-effects model. That does not matter so much for your particular question though.

I think your option (1) involves fitting separate models on subsets of the data, and I recommend against that because there is a lot of benefit from having one model where estimates benefit from considering all of the data.

In option (2), you indeed would be taking interactions into account if you do the comparisons of one factor separately for combinations of the other two factors.

But first, I'd suggest doing this graphically so you can see what is happening, with that 3-way interaction; e.g.,

emmeans::emmip(model, Time ~ pitch_level | Group)

which will create two panels (for Group) and in each panel, we get 3 curves (by Time) for the pitch levels.

Next, you can get numerical results. For instance, this code:

emm <- emmeans(model, ~ Group | Time * pitch_level)
confint(emm)
pairs(emm)

... obtains estimated marginal means (on the logit scale) and comparisons thereof, separately for each combination of Time and pitch_level. You can proceed with other things like

contrast(emm, "consec", simple = "Time")

which would do simple comparisons of consecutive times.

All this is on the logit scale. You can estimate things on the probability scale instead by adding type = "response" to the emmeans call, or indeed to any of the other calls. For the comparisons and contrasts, you will get odds ratios rather than differences.