# Estimated marginal means versus model-derived interaction effects

I'm getting confused with the results of experimental data with two groups and three measurements. Imagine a mixed-model in lme4, where I am interested in the effect of the interaction group*measurement. My model is a random-intercept model comparable to model1 <- lme(outcome ~ group + measurement + group:measurement + baseline_outcome + (1|ID).

My research question is: Is there any interaction between group and measurement time?

Now my model shows that there indeed is an interaction between group and measurement1, but not for group and measurement2 and group and measurement3.

If I use the emmeans package (emmeans(model1, pairwise ~ Group|measurement, infer=TRUE)), I get the exactly same results for the coefficients as I do with the multiplicative interaction of my initial lmer model. However, now there is no evidence for any group differences at measurement1 (same differences, different SEs, different p-values).

Regarding my research question, which term should I interpret? The multiplicative interaction or rather the results of the estimated marginal means? If the latter, what for do I need the multiplicative interaction coefficients anyway?

EDIT: Here are the formulas and models. Thanks for pointing this out.

lme4 model model1 <- lme(outcome ~ group + measurement + baseline_outcome + group:measurement + (1|ID)

Here are the relevant outputs (not for measurement main effect and baseline-outcome):

group1: Estimate = -3.33, SE = 0.61, df = 428, t-value = -1.99, p-value = 0.043

measurement 2 * group1: Estimate = 0.88, SE = 0.59, df = 140.24, t-value = 1.09, p-value = 0.30

measurement 3 * group1: Estimate = 0.86, SE = 0.58, df = 141.36, t-value = 1.06, p-value 0.33

So the effect of measurement 1:group1 = -3.33, for measurement2:group1 = -3.33+0.88 = -2.45, for measurement3:group1 = -3.33 + 0.86 = -2.47

emmeans model: emmeans (model1, pairwise ~ group|measurement, infer=TRUE, adjust = "none")

contrast estimate SE df t ratio p value
group0-group1 (measurement 1) -3.33 0.61 441 1.93 0.054
group0-group1 (measurement 2) -2.45 0.64 488 0.94 0.37
group0-group1 (measurement 3) -2.47 0.64 483 0.97 0.35
• It would help if you could edit the question to show the results of the different reports. From what you say so far, it might just be that emmeans is correcting for multiple comparisons while the other report isn't. Without seeing the reports, however, it's hard to know for sure.
– EdM
Commented Nov 9, 2023 at 16:15
• In addition to the answer from Robert Long, you should read this page for a discussion of the difficulties in defining degrees of freedom for a mixed model, and the emmeans vignette describing its options for lme models.
– EdM
Commented Nov 9, 2023 at 18:27

Interaction terms in linear models assess if group effects differ across measurements. A significant interaction suggests the group's influence on outcomes varies at different times.

emmeans calculates adjusted average group means for each measurement, considering other factors in the model. This adjustment can lead to different significance levels compared to the interaction term due to refined standard errors and p-values which are adjusted for multiple testing. This would not appear to be the reason in yhour case since you have used adjust=none.

One possibility is that the emmeans function computes standard errors and confidence intervals based on the marginal distribution of the EMMs, which takes into account the uncertainty in the estimation of the fixed effects, as well as the random effects structure of the model. The lmer model, however, tests the interaction terms based on the conditional distribution given the random effects.

For research questions on interactions, consider the multiplicative interaction to determine if an effect exists between group and measurement time. The multiplicative interaction coefficients indicate whether there is something to look at, while the EMMs tell you what the effect looks like.

If there is a significant interaction, further investigation can be achieved with EMMs to see where those differences lie. But if the interaction is not significant, this often suggests that the main effects of group and measurement time may be sufficient to explain the outcome variable. In practice, researchers often report both the interaction test results and the EMMs to provide a complete picture of the effects.

• Thanks a lot for your help! I directly used measurement time and group as factors, so I'm able to see at which measurement timepoint statistically significant differences arise. If I understand this correctly, the differences between the models could be because of 1) "wrong" coding of emmeans (adjusting for multiple testing while the interaction in the model doesnt; however, I thought i modelled this out with adjust=none) or 2) different adjustment, with emmeans considering other factors in the model (shouldnt the interaction adjust for these, too?)? Could there be other reasons? Thanks again. Commented Nov 9, 2023 at 18:01
• Ahh, I missed that ! I have updated the answer just now. Commented Nov 9, 2023 at 18:14
• @EdM and Robert Long, thanks a lot for your fast and helpful answers! I just used Satterthwaite's df approximation for the emmeans model and the differences still occur (even though now for measurement1 the p-values and SEs match between the multiplicative interaction and the emmeans; the difference in SE and p-values in measurement2 and measurement3 still remain). So this may be an issue with regards to the marginal distribution of the EMMs vs. the conditional distribution in the lmer model. Maybe someone else has another possible solution. Anyway: Any advices on which model is preferable? Commented Nov 9, 2023 at 18:57
• You'rte welcome. I think it's hard to choose. you might consider reporting results from both. Commented Nov 9, 2023 at 19:03
• I've just read in a recent high impact journal paper that "where no interaction was detected, effect of study arm was estimated by least square means". Can anyone relate to this? Interpret the multiplicative interaction term, and if it's not significant, estimate the effects of the treatment group via emms? Apart from that, I'm still struggling with reporting the multiplicative interaction as main analysis, as it's the difference in measurement times regarding the effect of group. Somehow it seems more logical to include the EMM comparisons per timepoint to me. Commented Nov 10, 2023 at 10:25