# Which statistical test to use (repeated measures ANOVA?)

I am a student conducting a clinical trial for my thesis. We have two independent variables: (1) time point (pre-test, post-test) and (2) control v. experimental group.

We'll be measuring about 6 or 7 continuous outcome variables (repeated measures at times one and two) to see if the experimental group benefits from the intervention.

I'm pretty rusty in statistics at this point and could use a starting point.

• How do you define "benefit"? Does it mean, e.g., one outcome variable is better, or all outcome variables are better, or some onjective function computed from the outcome varaible sis better, or ... Commented Jan 27, 2022 at 10:54

Yes, you could use repeated measures Anova (RMA). HOWEVER, but I would suggest to use mixed effects models with patient id as random effects instead (RMA is a type of mixed effects models). See this answer here going into more detail: https://stats.stackexchange.com/a/237294/343024.

To demonstrate how I would proceed here in R, let's assume your response is y, you have independent variables Time (pre, post) and Group (Control, Treat). All study participants have a unique id and have y recorded at both pre and post:

Create the dataset dd:

dd<-data.frame(id=rep(c(1:6000),2),
Time=rep(c("pre","post"), each=6000),
Group=rep(rep(c("Treat","Control"),2), each=3000),
y=c(rnorm(3000,90,5), rnorm(3000,87,5), rnorm(3000,70,5), rnorm(3000,88,5)))


Make pre reference level:

dd$Time=relevel(as.factor(dd$Time), ref="pre")


par(mfrow=c(1,1))
boxplot(y~Time+Group, dd)


Now fit the repeated measures Anova:

maov<- aov(y ~ Group*Time + Error(id), dd)
summary(maov)

Error: id
Df Sum Sq Mean Sq
Group  1 125676  125676

Error: Within
Df Sum Sq Mean Sq F value              Pr(>F)
Group          1  40480   40480    1635 <0.0000000000000002 ***
Time           1 272975  272975   11029 <0.0000000000000002 ***
Group:Time     1 336772  336772   13606 <0.0000000000000002 ***
Residuals  11995 296894      25


As you can see, your response y differs (P<0.0001) for variables Group, Time, and in particular the interaction term Group:Time (indicating that y changes differently between Control and Treat).

Now we can fit a mixed effects model using R-paclage lme4:

me1<-lmer(y~Time+Group+Group:Time+(1|id), dd)
summary(me1)

Linear mixed model fit by REML ['lmerMod']
Formula: y ~ Time + Group + Group:Time + (1 | id)
Data: dd

REML criterion at convergence: 72568.4

Scaled residuals:
Min      1Q  Median      3Q     Max
-3.9053 -0.6769  0.0024  0.6698  3.7353

Random effects:
Groups   Name        Variance Std.Dev.
id       (Intercept)  0.00    0.000
Residual             24.75    4.975
Number of obs: 12000, groups:  id, 6000

Fixed effects:
Estimate Std. Error  t value
(Intercept)          86.91326    0.09083  956.880
Timepost              1.05620    0.12845    8.222
GroupTreat            3.15322    0.12845   24.548
Timepost:GroupTreat -21.19032    0.18166 -116.648


You can see straight away from:

1. the coefficient for Time, that your response y in Control (reference level in variable Group) increased by 1.05620 from pre (reference level in variable Time) to post
2. the coefficient for Group, that your response y in Treat at reference-Time pre was 3.15322 higher than in Control
3. the change from pre to post in Treat was 21.19032 lower than Control, which is your difference of interest between both groups.

Be aware that per default, R-package lme4 does not provide p-values (for good reasons). However, there are ways to obtain them by using for example package lmerTest:

library(lmerTest)
anova(me1)
Type III Analysis of Variance Table with Satterthwaite's method
Sum Sq Mean Sq NumDF DenDF F value                Pr(>F)
Time       272975  272975     1 11996   11029 < 0.00000000000000022 ***
Group      166147  166147     1 11996    6713 < 0.00000000000000022 ***
Time:Group 336772  336772     1 11996   13607 < 0.00000000000000022 ***


• How you model depends entirely on what "pre" means. If it is a random measurement of primary interest then putting it in the mixed effects model as above makes sense. If it is an "equalizer" (adjustment variable) then a univariate model is needed with a baseline covariate. The latter is typically more appropriate. Commented Jan 27, 2022 at 12:52
• @Frank: good point I have not considered. I simply assumed that response "y" is measured within a certain time window prior to intervention, and then again after a certain time following the intervention. I might be completely wrong here, but would a univariate model with a baseline covariate not be asking a slightly different question? I.e. "what is the difference in mean change from pre to post between groups" (mixed effects) versus "what is the mean difference between groups at post, adjusted for pre-test score" (univariate model with baseline as covariate)? Commented Jan 27, 2022 at 14:08
• It's best not to use the concept of "difference" when referring to within-person difference (due to regression to the mean and a host of other problems). The key question is usually this: Do two subjects who start at the same point (same baseline value of Y) except for having different interventions end up at the same point? That makes it clear that random effects are not the way to go, usually. Commented Jan 28, 2022 at 14:20