2
$\begingroup$

I am working with a colleague's data from a research project comparing two groups of learners on a test of language ability. Data are repeated measures, between-subjects. Participants in each group (M or C) took the same test at time A and later at time B.

Possible scores on tests could be 0-5, but in reality ranged from 2-5. Scores had to be one of 2, 3, 4, or 5 (which means this is probably more appropriately an ordinal variable, but that's a separate issue.)

Our goal is to test whether there are significant differences between the groups on learning, as measured by any gains on the test score time A to time B. As such I fit a multilevel regression model with an interaction term between time and group, with random intercepts for subjects.

Below I simulate our actual data with the same number of participants per group and approximate probabilities of the score distributions per group / time. The result is a very similar distribution of scores for the two groups and a similar model with significant interaction in the same direction as our data.

set.seed(100)
group <- c(rep('M', 18), rep('C', 23), rep('M', 18), rep('C', 23))
time <- c(rep('A', 18), rep('A', 23), rep('B', 18), rep('B', 23))
subject <- c(1:18, 19:41, 1:18, 19:41)
MA <- sample(c(2,3,4,5), prob = c(.43,.47,.08,0), size = 18, replace = T)
MB <- sample(c(2,3,4,5), prob = c(0,.17,.69,.13), size = 18, replace = T)
CA <- sample(c(2,3,4,5), prob = c(.28,.39,.33,0), size = 23, replace = T)
CB <- sample(c(2,3,4,5), prob = c(0,.23,.79,0), size = 23, replace = T)
sim_dat <- tibble(group, time, score = c(MA, CA, MB, CB), subject)

m1 <- lmer(score ~ group*time + (1|subject), data = sim_dat)
summary(m1)

For brevity here are the fixed effects from the model:

Fixed effects:
             Estimate Std. Error      df t value Pr(>|t|)    
(Intercept)    2.9565     0.1210 77.9767  24.438  < 2e-16 ***
groupM        -0.4010     0.1826 77.9767  -2.196   0.0311 *  
timeB          0.8261     0.1696 38.9999   4.871 1.88e-05 ***
groupM:timeB   0.6739     0.2560 38.9999   2.633   0.0121 *  

My understanding of how to interpret the coefficients is that the intercept reflects the baseline levels (in this case, Group C at Time A).

The term groupM then considers the difference between the intercept and the change from group C to M. So, practically, the comparison of Group M to Group C at time A. This indicates Group M had a lower score at time A than Group C.

This makes sense when looking at the pairwise comparisons, which also indicate no differences between the groups at time B (which is not shown in the model summary):

pairs(emmeans(m1, ~group | time))

time = A:
 contrast estimate    SE df t.ratio p.value
 C - M       0.401 0.183 78   2.196  0.0311

time = B:
 contrast estimate    SE df t.ratio p.value
 C - M      -0.273 0.183 78  -1.495  0.1390

Degrees-of-freedom method: kenward-roger 

The term timeB then considers the difference between the intercept and the change from time A to time B. So, practically, the comparison of Group C Time A to Group C Time B. This indicates Group C had a higher score at time B than time A.

This also makes sense when looking at the pairwise comparisons, which also show that Group M had a higher score at Time B versus Time A (again, this contrast is not shown in the model summary):

pairs(emmeans(m1, ~time | group))

group = C:
 contrast estimate    SE df t.ratio p.value
 A - B      -0.826 0.170 39  -4.871  <.0001

group = M:
 contrast estimate    SE df t.ratio p.value
 A - B      -1.500 0.192 39  -7.824  <.0001

Degrees-of-freedom method: kenward-roger 

The interaction term is where I lose the script. My understanding is that groupM:timeB is compared against the baseline, so would that mean Group M Time B against Group C Time A?

However, what it seems to be reporting is the overall magnitude of the difference between Time A and Time B for Group M versus the magnitude of the difference between Time A and Time B for Group C (which, of course, is exactly what we want to test in this data!). If that is the case, then does the estimate mean that Group M was predicted on average to have a higher increases from Time A to Time B (of .67 score units) when compared to Group C?

Doing a "pairwise of pairwise" comparison seems to suggest that is the case, since I get the same terms from the model:

contrast(emmeans(m1, ~group*time), interaction = 'pairwise')

 group_pairwise time_pairwise estimate    SE df t.ratio p.value
 C - M          A - B            0.674 0.256 39   2.633  0.0121

Degrees-of-freedom method: kenward-roger 

So, my question is, am I just fumbling the interpretation of the labels for the interaction term model? Is my understanding of the contrasts incorrect for interacting factors? I think I understand now what the model is showing (i.e., increased gains for group M versus group C), but I still don't fully understand how the model is doing this and how I am to interpret the coefficient straight from the model (in fact, I didn't think that was possible!).

$\endgroup$
0

1 Answer 1

2
$\begingroup$

The interaction term is where I lose the script. My understanding is that groupM:timeB is compared against the baseline, so would that mean Group M Time B against Group C Time A?

As you discovered, that is NOT what the interaction term represents. With this type of data coding, when neither predictor is at its reference level, it's the extra difference beyond what you would have estimated based simply on the values of lower-level coefficients. That interpretation has the advantage of extending to even higher-level interactions.

Trying to interpret the coefficients directly from a model summary that includes interactions can lead to lots of difficulties. Besides the one that you found, simply changing the reference level of one predictor can change the estimate of the coefficient for an interacting predictor and thus the apparent "significance" of what some call that interacting predictor's "main effect" coefficient.

Your approach, proceeding to post-modeling tools like those in emmeans, leads to much less confusion. That was a good choice.

$\endgroup$
3
  • $\begingroup$ Thank you @EdM for the reply and clarification! Is it correct for me to interpret the interaction term as the effect of group M to group C AND the effect of time A to time B? I'm still a bit unclear as to how to phrase this in a manner similar to comparing either the group or time terms on their own. $\endgroup$
    – sskalicky
    Apr 29, 2023 at 4:57
  • 1
    $\begingroup$ @sskalicky with the default coding in R, it's simplest to build up from the lowest levels to progressively higher ones. The intercept is the predicted outcome when all predictors are at reference levels (0 if continuous). An individual coefficient for a predictor is the difference from that when all other predictors are at reference. A two-way interaction is the extra difference beyond what you would predict based on those 2 predictors' individual coefficients (with all other predictors at reference). And so on up the interaction hierarchy. Other interpretations lead to confusion. $\endgroup$
    – EdM
    Apr 30, 2023 at 21:55
  • $\begingroup$ Ah ha, that makes so much more sense to me now, thank you for explaining! $\endgroup$
    – sskalicky
    May 2, 2023 at 0:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.