# When to use repeated measures versus ANOVA on differences

I conducted an experiment where students were divided between two experiment groups and two language levels. They all completed a pre-test and post-test. I have tried the following two approaches, but I am not sure which one is giving me the correct answer. I did a mixed ANOVA (within subjects test scores) and between subjects was the experiment group and language levels, using the Repeated measures ANOVA function on SPSS. I also did a two-way ANCOVA on the post-test with the pre-test as a co-variate the experiment and language groups were the between subjects factors. The problem is that the results do not seem to be the same. My groups were randomly allocated and there are no significant differences in their pre-test scores.

• There's actually extensive literature on this question; it hasn't been resolved by a long shot, and each situation may present its own challenge. For a bit of info see queryoverflow.gdn/query/… . A longer source that looks informative is homes.ori.org//keiths/Tips/Stats_GainScores.html . – rolando2 Jul 9 '18 at 18:36
• I removed the references to SPSS, which weren't really necessary to the question and which would tend to make the question off-topic on this site. – rolando2 Jul 9 '18 at 18:38
• – rolando2 Jul 9 '18 at 18:40
• It's been a while but I find it strange that repeated measures ANOVA of two time points with the same between subjects factors is any different from ANOVA with same between subjects factors of the difference score. The t-test analog of a repeated measures ANOVA is the paired t-tests which is just analysis of the difference score. I think your results should only be different if you did an ANCOVA controlling for the pretest with the same between subjects factors. – Heteroskedastic Jim Jul 9 '18 at 19:30
• I have changed my post. It is quite correct the ANOVA on the difference scores and the repeated measures gave the same results. It is the ANCOVA using the pre-test as a co-variate that gives me a different result. I am looking for a non-technical explanation that I can use to explain which method I should use. Thanks for all of the feedback. – Sonnette Smith Jul 15 '18 at 17:07

The test of the main hypotheses should be the same for both of these approaches. Keep in mind that the factors of interest are different. In the second model of change (gain) scores, you have three F-values: a main-effect each for experiment and language and the interaction between the two. In the former within/between repeated measures ANOVA, you have seven F-values: main effects for experiment, language, and time as well as three two-way interactions and one three-way interaction. The two-way interactions correspond to the main effects in the first model and the three-way interaction corresponds to the two-way interaction in the first model. For example, the time by experiment interaction is testing whether change over time differs between the groups. Thus, this is the main effect for experiment in the change score ANVOA.

Below demonstrates that these two models produce identical F-values for the three hypotheses tested by the change score model.

> ## Create data in long format (2 rows per student)
> set.seed(1)
> y <- rnorm(32)
> student <- c(1,1,2,2,3,3,4,4,5,5,6,6,7,7,8,8,9,9,
+              10,10,11,11,12,12,13,13,14,14,15,15,16,16)
> student <- factor(student)
> time <- c(rep(c(1,2),16))
> experiment <- c(rep(1,16),rep(2,16))
> language <- c(rep(1,8),rep(2,8),rep(1,8),rep(2,8))
>
> ## run repeasted measures ANOVA
> summary(aov(y ~ experiment*language*time + Error(student/time)))

Error: student
Df Sum Sq Mean Sq F value Pr(>F)
experiment           1  0.020  0.0197   0.017  0.899
language             1  0.216  0.2165   0.186  0.674
experiment:language  1  0.058  0.0579   0.050  0.827
Residuals           12 13.963  1.1635

Error: student:time
Df Sum Sq Mean Sq F value Pr(>F)
time                      1  1.300   1.300   2.073 0.1755
experiment:time           1  0.080   0.080   0.128 0.7272
language:time             1  2.100   2.099   3.349 0.0922 .
experiment:language:time  1  1.133   1.133   1.806 0.2038
Residuals                12  7.524   0.627
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

> ## Createdata with change (gain) score
> gain <- y[time==2] - y[time==1]
> experiment <- c(rep(1,8),rep(2,8))
> language <- c(rep(1,4),rep(2,4),rep(1,4),rep(2,4))
>
> ## run ANOVA on change scores
> summary(aov(gain ~ experiment*language))
Df Sum Sq Mean Sq F value Pr(>F)
experiment           1  0.160   0.160   0.128 0.7272
language             1  4.199   4.199   3.349 0.0922 .
experiment:language  1  2.265   2.265   1.806 0.2038
Residuals           12 15.047   1.254
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1


You can see in the above that the F for the experiment by language interaction effect in the change score ANOVA is equal to the time by experiment by language F from the repeated measure ANOVA (both are 1.806 with 1,12 degrees for freedom and a p-value of 0.2038).