# Different results with different methods of conducting RM ANOVA in R

I have a repeated measures design, one continious outcome "E_R", two categorical predictors "S" and "I". In my design, every participant has been exposed to every level of S and every level of I in random orders (so there's no time variable here).

I have my data in long format with 4 collumns e.g.:

ID - "I" - "S" - "E_R"

1 - 1 - c - -32.6

1 - 1 - d - 6.5

1 - 1 - e - -4.2

1 - 2 - c - -8.4

...

I tried performing the analysis with thee methods I found online. For two methods I get very similar results (although not exactly the same). But with the third method, I get different results (the conclusions are the same, but the F values are very different).

Method 1:

model <- aov(E_R ~ S * I + Error(ID), data = DF_long)

summary(model)

Method 2 (with lme4 and lmerTest):

model_mixed <- lmer(E_R ~ S * I + (1|ID), data = DF_long)

anova(model_mixed)

Method 3 (with rstatix):

res.aov <- anova_test( data = DF_long, dv = E_R, wid = ID, within = c(S, I) )

anova_table <- get_anova_table(res.aov)

anova_table

Here are the results with each method:

Method 1 results:

"S" Df = 4, Sum Sq = 151817, Mean Sq = 37954, F value = 32.81, p < 0.0001

"I" Df = 3, Sum Sq = 1097276, Mean Sq = 365759, F value = 316.15, p < 0.0001

"S":"I" Df = 12, Sum Sq = 11668, Mean Sq = 972, F value = 0.84, p = 0.609

Residuals Df = 985, Sum Sq = 1139567, Mean Sq = 1157

Method 2 results:

"S" Sum Sq = 145155, Mean Sq = 36289, NumDF= 4, DenDF = 985.06, F value = 31.3661, P < 0.0001

"I" Sum Sq = 1093943, Mean Sq = 364648, NumDF= 3, DenDF = 985.05, F value = 315.1827, P < 0.0001

"S":"I" Sum Sq = 11643, Mean Sq = 970, NumDF= 12, DenDF = 985.07, F value = 0.8386, P = 0.6104

Method 3 results:

"S" DFn = 2.90 , DFd = 113.02, F = 23.886, p < 0.0001, ges = 0.05

"I" DFn = 1.36, DFd = 53.12, F = 94.788, p < 0.0001, ges = 0.305

"S":"I" DFn = 7.68 , DFd = 299.67, F = 1.344, p = 0.224, ges = 0.007

I'm trying to understand why the results are different, and which one is the correct result. But I can't find anything... Can anyone please help me?

• Is it possible that you did not convert the predictors to factor variables before computing the results? It might be possible that Variable I with levels 1, 2, ... is seen as numerical, which would lead to erroneous results with the first two approaches. Commented Jan 24, 2023 at 14:34
• @David I did convert the predictors to factor variables (I just checked). So that's not it...
– Y45H
Commented Jan 24, 2023 at 14:53
• I don't use rstatix, but a quick look at the manual suggests that you need to specify a random-effect term similar to that you used in Method 1. See the example in the anova_test help section on the "formula" argument for "Within-SsANOVA(repeatedmeasuresANOVA)."
– EdM
Commented Jan 24, 2023 at 14:55
• @EdM it doesn't seem to be working. When I use the formula approach I get: Error in Error(ID) : could not find function "Error". The manual is inconsistent with the use of the Error term in the formular it sometimes uses Error() and sometimes error(), I tried both and none of these worked.
– Y45H
Commented Jan 24, 2023 at 15:03

I'm not a great fan of rstatix, which you use in Method 3. It's supposed to provide a simpler interface to standard R functions, like those you used in Methods 1 and 2, but I fear that there can be hidden assumptions in the implementation that can lead to this type of discrepancy. And, as you note in a comment, the help file and the actual function don't necessarily agree. Based on what I've seen in answering questions on this site, that package and the similar survminer package, both intended to fit nicely into the "tidyverse," have glitches that recommend against relying on them routinely.