# A mixed-effects model for repeated measurements vs multiple time point-wise comparisons with a simpler test

I have a pretty standard situation of a study in which repeated measurements are taken from the same individuals. There are two factors: "Group" (with 25 individuals in each of two groups) and "Day" (time is treated here as a categorical variable). To keep things simple, let's consider only two time points, Day 1 and Day 2. When working in R, the data would look as follows (ID - subjects' IDs; Group - labels for the groups; Day - factor indicating the day of sampling, with 2 levels; BW - body weight, kg):

 dat
ID Group   Day       BW
1   ID1     A Day 1 2333.231
2   ID2     A Day 1 2615.744
3   ID3     A Day 1 2282.484
4   ID4     A Day 1 2796.806
5   ID5     A Day 1 2262.759
6   ID6     A Day 1 2520.216
7   ID7     A Day 1 2606.598
8   ID8     A Day 1 2617.347
9   ID9     A Day 1 2439.651
10 ID10     A Day 1 2515.900
11 ID11     B Day 1 2692.253
12 ID12     B Day 1 2208.707
13 ID13     B Day 1 2343.652
14 ID14     B Day 1 2564.080
15 ID15     B Day 1 2411.044
16 ID16     B Day 1 2774.001
17 ID17     B Day 1 2634.651
18 ID18     B Day 1 2514.433
19 ID19     B Day 1 2198.449
20 ID20     B Day 1 2505.220
21  ID1     A Day 2 2314.214
22  ID2     A Day 2 2302.396
23  ID3     A Day 2 2319.029
24  ID4     A Day 2 2533.612
25  ID5     A Day 2 2290.300
26  ID6     A Day 2 2168.727
27  ID7     A Day 2 2466.597
28  ID8     A Day 2 2223.379
29  ID9     A Day 2 2441.762
30 ID10     A Day 2 2288.917
31 ID11     B Day 2 1984.846
32 ID12     B Day 2 2702.819
33 ID13     B Day 2 2793.834
34 ID14     B Day 2 2563.337
35 ID15     B Day 2 2666.664
36 ID16     B Day 2 2399.159
37 ID17     B Day 2 2586.255
38 ID18     B Day 2 2193.912
39 ID19     B Day 2 2797.592
40 ID20     B Day 2 3043.074


Here is a graphical representation of these data (data points coming from the same subject are connected with dashed lines to make it easier to understand the structure of this dataset):

In order to test the effects of Group and Day, I could fit a mixed-effects model using e.g. the nlme package for R:

# Fit the model:
M <- lme(BW ~ Day * Group, random = ~ 1 | ID, data = dat)

# check the significance of effects:
anova(M)
numDF denDF  F-value p-value
(Intercept)     1    18 5564.085  <.0001
Day             1    18    0.326  0.5753
Group           1    18    2.849  0.1087
Day:Group       1    18    3.631  0.0728


Thus, according to the fitted mixed-effects model (which was adequate for these data - diagnostics were run but are not presented here), neither of the examined factors (Day and Group) are affecting the response variable; also, there is no interaction between the two factors.

This is the type of analysis that I would do for such a dataset if I were asked to. However, in my organisation many people have no idea about the mixed-effects models. What they would typically do is applying a bunch of t-tests (or similar tests) to detect the effect of the "Group" on each of the sampling dates. For example, for the data shown above one would conduct a t-test for Day 1 and another t-test for Day 2, getting the following results:

Day 1: P = 0.271

Day 2: P < 0.001

Thus, they would claim that there was a significant Group effect on Day 2. I tried to explain that this result would not be correct because of the presence of correlation in data, which originates from the repeated measurements made on the same subjects. However, a colleague of mine asked a question that I could not answer easily. He said:

"Ok, the observations are correlated, I get that. But for now, forget about the fact that we have data from Day 1 and suppose that there are data only from Day 2. Observations in Group A and Group B are independent from each other, and so we are allowed to apply to a t-test or something similar. When we do apply a t-test [as shown above], we get a significant Group effect. How should we then treat this result?"

And this is exactly the point were I got stuck. Indeed, if one has only the information from Day 2 and does a simple t-test, one gets a very different (and, in principle, justified) conclusion than the one obtained with the mixed effects model. Which method of analysis is to trust then? Is the Group effect real?

I feel like I am missing some important piece for justification of the use of mixed model. Any hint would be highly appreciated.

• I don't see how there is any contradiction or paradox here. The question you asked of the data with your mixed model ("Is there a Group effect on average across Days 1 and 2?") is completely different from the question your colleague asked using t-tests ("Is there a Group effect on Day 2?"). Why would you expect different questions to yield the same answer? – Jake Westfall Sep 6 '13 at 19:20
• Dear Jake, thank you for you comment, it's very helpful. I got it: indeed, the questions being asked are different, and so can be the answers. Beautiful! – user23854 Sep 8 '13 at 18:35

Taking your second point first, your analysis of Day looked at the aggregate across days and there is no Day effect on average. There might be one on Day 2 but you really should have a justification for believing Day 2 more than other days.

Point 1, that Day 1 isn't significant while Day 2 does show an effect is a meaningless point to make. Ignoring the correlation and analysis techniques, even if what your colleagues claim is true, it's not useful. The implied argument is that the effect of group in Day 1 is different from Day 2 and that wasn't tested. That's what your interaction tested and it's not significant.

Finally, from the tenor of this report it sounds like there's a lot of being hung up on what significant and what's not. For example, if Day 1 and Day 2 effects are both in the same direction but one is significant and one is not are they really contradictory? Think about that.

• John, thank you for your answer, especially for the link to Gelman and Stern's paper. – user23854 Sep 8 '13 at 18:47

This may raise more questions for you than it answers, but consider that you can directly test A2 vs B2 by looking directly at the coefficients (plus their standard error) of your model and linearly combining them

> summary(M)
Linear mixed-effects model fit by REML
...
Fixed effects: BW ~ Day * Group
Value Std.Error DF  t-value p-value
(Intercept)  2499.0736  66.30508 18 37.69053  0.0000
DayD2        -164.1803  93.76955 18 -1.75089  0.0970
GroupB        -14.4246  93.76955 18 -0.15383  0.8795
DayD2:GroupB  252.6805 132.61017 18  1.90544  0.0728


In particular, the difference in group means on day 2 between A and B is 238.26, which when tested

> anova(M, L=c(0,0,1,1))
F-test for linear combination(s)
GroupB DayD2:GroupB
1            1
numDF denDF  F-value p-value
1     1    18 6.456002  0.0205

> t.test(subset(dat, Day=="D2" & Group=="B")$BW, subset(dat, Day=="D2" & Group=="A")$BW)
...
t = 2.2832, df = 11.332, p-value = 0.04264
...
mean of x mean of y
2573.149  2334.893

• I think I'm missing the point of this, could you clarify? – John Sep 8 '13 at 20:51
• Sorry, you've lost me: what's the code to get to your model? – DavidP Sep 9 '13 at 16:36