I have a pretty standard situation of a study in which repeated measurements are taken from the same individuals. There are two factors: "Group" (with 25 individuals in each of two groups) and "Day" (time is treated here as a categorical variable). To keep things simple, let's consider only two time points, Day 1 and Day 2. When working in R, the data would look as follows (ID - subjects' IDs; Group - labels for the groups; Day - factor indicating the day of sampling, with 2 levels; BW - body weight, kg):

     ID Group   Day       BW
 1   ID1     A Day 1 2333.231
 2   ID2     A Day 1 2615.744
 3   ID3     A Day 1 2282.484
 4   ID4     A Day 1 2796.806
 5   ID5     A Day 1 2262.759
 6   ID6     A Day 1 2520.216
 7   ID7     A Day 1 2606.598
 8   ID8     A Day 1 2617.347
 9   ID9     A Day 1 2439.651
 10 ID10     A Day 1 2515.900
 11 ID11     B Day 1 2692.253
 12 ID12     B Day 1 2208.707
 13 ID13     B Day 1 2343.652
 14 ID14     B Day 1 2564.080
 15 ID15     B Day 1 2411.044
 16 ID16     B Day 1 2774.001
 17 ID17     B Day 1 2634.651
 18 ID18     B Day 1 2514.433
 19 ID19     B Day 1 2198.449
 20 ID20     B Day 1 2505.220
 21  ID1     A Day 2 2314.214
 22  ID2     A Day 2 2302.396
 23  ID3     A Day 2 2319.029
 24  ID4     A Day 2 2533.612
 25  ID5     A Day 2 2290.300
 26  ID6     A Day 2 2168.727
 27  ID7     A Day 2 2466.597
 28  ID8     A Day 2 2223.379
 29  ID9     A Day 2 2441.762
 30 ID10     A Day 2 2288.917
 31 ID11     B Day 2 1984.846
 32 ID12     B Day 2 2702.819
 33 ID13     B Day 2 2793.834
 34 ID14     B Day 2 2563.337
 35 ID15     B Day 2 2666.664
 36 ID16     B Day 2 2399.159
 37 ID17     B Day 2 2586.255
 38 ID18     B Day 2 2193.912
 39 ID19     B Day 2 2797.592
 40 ID20     B Day 2 3043.074

Here is a graphical representation of these data (data points coming from the same subject are connected with dashed lines to make it easier to understand the structure of this dataset):

enter image description here

In order to test the effects of Group and Day, I could fit a mixed-effects model using e.g. the nlme package for R:

# Fit the model:
M <- lme(BW ~ Day * Group, random = ~ 1 | ID, data = dat)

# check the significance of effects:
            numDF denDF  F-value p-value
(Intercept)     1    18 5564.085  <.0001
Day             1    18    0.326  0.5753
Group           1    18    2.849  0.1087
Day:Group       1    18    3.631  0.0728

Thus, according to the fitted mixed-effects model (which was adequate for these data - diagnostics were run but are not presented here), neither of the examined factors (Day and Group) are affecting the response variable; also, there is no interaction between the two factors.

This is the type of analysis that I would do for such a dataset if I were asked to. However, in my organisation many people have no idea about the mixed-effects models. What they would typically do is applying a bunch of t-tests (or similar tests) to detect the effect of the "Group" on each of the sampling dates. For example, for the data shown above one would conduct a t-test for Day 1 and another t-test for Day 2, getting the following results:

Day 1: P = 0.271

Day 2: P < 0.001

Thus, they would claim that there was a significant Group effect on Day 2. I tried to explain that this result would not be correct because of the presence of correlation in data, which originates from the repeated measurements made on the same subjects. However, a colleague of mine asked a question that I could not answer easily. He said:

"Ok, the observations are correlated, I get that. But for now, forget about the fact that we have data from Day 1 and suppose that there are data only from Day 2. Observations in Group A and Group B are independent from each other, and so we are allowed to apply to a t-test or something similar. When we do apply a t-test [as shown above], we get a significant Group effect. How should we then treat this result?"

And this is exactly the point were I got stuck. Indeed, if one has only the information from Day 2 and does a simple t-test, one gets a very different (and, in principle, justified) conclusion than the one obtained with the mixed effects model. Which method of analysis is to trust then? Is the Group effect real?

I feel like I am missing some important piece for justification of the use of mixed model. Any hint would be highly appreciated.

  • $\begingroup$ I don't see how there is any contradiction or paradox here. The question you asked of the data with your mixed model ("Is there a Group effect on average across Days 1 and 2?") is completely different from the question your colleague asked using t-tests ("Is there a Group effect on Day 2?"). Why would you expect different questions to yield the same answer? $\endgroup$ Sep 6, 2013 at 19:20
  • $\begingroup$ Dear Jake, thank you for you comment, it's very helpful. I got it: indeed, the questions being asked are different, and so can be the answers. Beautiful! $\endgroup$
    – user23854
    Sep 8, 2013 at 18:35

2 Answers 2


Taking your second point first, your analysis of Day looked at the aggregate across days and there is no Day effect on average. There might be one on Day 2 but you really should have a justification for believing Day 2 more than other days.

Point 1, that Day 1 isn't significant while Day 2 does show an effect is a meaningless point to make. Ignoring the correlation and analysis techniques, even if what your colleagues claim is true, it's not useful. The implied argument is that the effect of group in Day 1 is different from Day 2 and that wasn't tested. That's what your interaction tested and it's not significant.

Finally, from the tenor of this report it sounds like there's a lot of being hung up on what significant and what's not. For example, if Day 1 and Day 2 effects are both in the same direction but one is significant and one is not are they really contradictory? Think about that.

  • $\begingroup$ John, thank you for your answer, especially for the link to Gelman and Stern's paper. $\endgroup$
    – user23854
    Sep 8, 2013 at 18:47

This may raise more questions for you than it answers, but consider that you can directly test A2 vs B2 by looking directly at the coefficients (plus their standard error) of your model and linearly combining them

> summary(M) 
Linear mixed-effects model fit by REML 
Fixed effects: BW ~ Day * Group  
                 Value Std.Error DF  t-value p-value 
(Intercept)  2499.0736  66.30508 18 37.69053  0.0000 
DayD2        -164.1803  93.76955 18 -1.75089  0.0970 
GroupB        -14.4246  93.76955 18 -0.15383  0.8795 
DayD2:GroupB  252.6805 132.61017 18  1.90544  0.0728 

In particular, the difference in group means on day 2 between A and B is 238.26, which when tested

> anova(M, L=c(0,0,1,1)) 
F-test for linear combination(s) 
      GroupB DayD2:GroupB  
           1            1  
  numDF denDF  F-value p-value 
1     1    18 6.456002  0.0205 

> t.test(subset(dat, Day=="D2" & Group=="B")$BW, subset(dat, Day=="D2" & Group=="A")$BW) 
t = 2.2832, df = 11.332, p-value = 0.04264 
mean of x mean of y  
 2573.149  2334.893  
  • $\begingroup$ I think I'm missing the point of this, could you clarify? $\endgroup$
    – John
    Sep 8, 2013 at 20:51
  • $\begingroup$ Sorry, you've lost me: what's the code to get to your model? $\endgroup$
    – DavidP
    Sep 9, 2013 at 16:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.