0
$\begingroup$

I have a repeated measures design, one continious outcome "E_R", two categorical predictors "S" and "I". In my design, every participant has been exposed to every level of S and every level of I in random orders (so there's no time variable here).

I have my data in long format with 4 collumns e.g.:

ID - "I" - "S" - "E_R"

1 - 1 - c - -32.6

1 - 1 - d - 6.5

1 - 1 - e - -4.2

1 - 2 - c - -8.4

...

I tried performing the analysis with thee methods I found online. For two methods I get very similar results (although not exactly the same). But with the third method, I get different results (the conclusions are the same, but the F values are very different).

Method 1:

model <- aov(E_R ~ S * I + Error(ID), data = DF_long)

summary(model)

Method 2 (with lme4 and lmerTest):

model_mixed <- lmer(E_R ~ S * I + (1|ID), data = DF_long)

anova(model_mixed)

Method 3 (with rstatix):

res.aov <- anova_test( data = DF_long, dv = E_R, wid = ID, within = c(S, I) )

anova_table <- get_anova_table(res.aov)

anova_table

Here are the results with each method:

Method 1 results:

"S" Df = 4, Sum Sq = 151817, Mean Sq = 37954, F value = 32.81, p < 0.0001

"I" Df = 3, Sum Sq = 1097276, Mean Sq = 365759, F value = 316.15, p < 0.0001

"S":"I" Df = 12, Sum Sq = 11668, Mean Sq = 972, F value = 0.84, p = 0.609

Residuals Df = 985, Sum Sq = 1139567, Mean Sq = 1157

Method 2 results:

"S" Sum Sq = 145155, Mean Sq = 36289, NumDF= 4, DenDF = 985.06, F value = 31.3661, P < 0.0001

"I" Sum Sq = 1093943, Mean Sq = 364648, NumDF= 3, DenDF = 985.05, F value = 315.1827, P < 0.0001

"S":"I" Sum Sq = 11643, Mean Sq = 970, NumDF= 12, DenDF = 985.07, F value = 0.8386, P = 0.6104

Method 3 results:

"S" DFn = 2.90 , DFd = 113.02, F = 23.886, p < 0.0001, ges = 0.05

"I" DFn = 1.36, DFd = 53.12, F = 94.788, p < 0.0001, ges = 0.305

"S":"I" DFn = 7.68 , DFd = 299.67, F = 1.344, p = 0.224, ges = 0.007

I'm trying to understand why the results are different, and which one is the correct result. But I can't find anything... Can anyone please help me?

Improve this question
$\endgroup$
4
  • $\begingroup$ Is it possible that you did not convert the predictors to factor variables before computing the results? It might be possible that Variable I with levels 1, 2, ... is seen as numerical, which would lead to erroneous results with the first two approaches. $\endgroup$
    – David
    Jan 24 at 14:34
  • $\begingroup$ @David I did convert the predictors to factor variables (I just checked). So that's not it... $\endgroup$
    – Y45H
    Jan 24 at 14:53
  • $\begingroup$ I don't use rstatix, but a quick look at the manual suggests that you need to specify a random-effect term similar to that you used in Method 1. See the example in the anova_test help section on the "formula" argument for "Within-SsANOVA(repeatedmeasuresANOVA)." $\endgroup$
    – EdM
    Jan 24 at 14:55
  • $\begingroup$ @EdM it doesn't seem to be working. When I use the formula approach I get: Error in Error(ID) : could not find function "Error". The manual is inconsistent with the use of the Error term in the formular it sometimes uses Error() and sometimes error(), I tried both and none of these worked. $\endgroup$
    – Y45H
    Jan 24 at 15:03

2 Answers 2

Reset to default
1
$\begingroup$

Methods 1 and 2 are well established, reliable ways to deal with repeated measures. Method 1 can be misleading if there are different numbers of cases in the treatment groups, but that's not a problem in your situation. The results of those two methods agree very closely, as they are essentially the same model specified in different ways.

I'm not a great fan of rstatix, which you use in Method 3. It's supposed to provide a simpler interface to standard R functions, like those you used in Methods 1 and 2, but I fear that there can be hidden assumptions in the implementation that can lead to this type of discrepancy. And, as you note in a comment, the help file and the actual function don't necessarily agree. Based on what I've seen in answering questions on this site, that package and the similar survminer package, both intended to fit nicely into the "tidyverse," have glitches that recommend against relying on them routinely.

Stick with the more established R functions.

An alternative for repeated measures that you didn't try, but might consider in future work without such a complete data set, is generalized least squares. See Chapter 7 of Frank Harrell's online notes for discussion of different ways to handle such data.

Improve this answer
$\endgroup$
1
$\begingroup$

Not sure about rstatix, but the first two approaches appear correct to me and produce equivalent results. rstatix is producing different numerator and denominator df.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.