0
$\begingroup$

I would like to know if it is necessary to correct the "overall" tests ofmain-effects/interaction effect for multiple comparisons in a 2x2 mixed-effects repeated-measures ANOVA.

Simple example: Imagine a model with a (between-subject) factor of "group" (2 levels), a (within-subject) factor of "time" (2 levels) and an interaction factor of group x time. If you just want to investigate the main-effects ("group" and "time", 2 tests) and the interaction-effect (1 test) at alpha=0.05 would you need to adjust alpha via (e.g.) Bonferroni to alpha=0.05/3?

Please note that I am not talking about posthoc tests following a significant interaction effect (which have to be corrected). I am just talking about what the significance threshold should be to determine if the interaction is significant in the first place.

I was always thinking that such a correction is not necessary (alpha=0.05 is sufficient) but was recently told that I absolutely need to correct for multiple comparisons on the factor-level already (alpha=0.05/3)...

Improve this question
$\endgroup$
1
$\begingroup$

If you want to control the type 1 error rate across the three null hypotheses, then an adjustment is necessary in the scenario you describe.

Improve this answer
$\endgroup$
  • $\begingroup$ But is that actually necessary in the case of the RM-ANOVA/is it normally done? Do you maybe have a reference/tutorial mentioning this? $\endgroup$ – Paul Aug 1 '19 at 13:13
  • $\begingroup$ It is, if you wish to control the familywise type 1 error. Multiple tests except in some very special cases (e.g. nested hypotheses) simply do. $\endgroup$ – Björn Aug 1 '19 at 13:19
  • $\begingroup$ OK. Can you point me to any reference (book/paper/tutorial/statistical package description) describing this in more detail? (Not trying to argue against you just would like to see an "official" reference stating that) $\endgroup$ – Paul Aug 1 '19 at 13:25
  • $\begingroup$ What about: en.m.wikipedia.org/wiki/… $\endgroup$ – Björn Aug 1 '19 at 13:35
  • $\begingroup$ Hmm... but that does not really address the issue of the RM-ANOVA? The only paragraph in there about ANOVA in general is: "Methods which rely on an omnibus test before proceeding to multiple comparisons. Typically these methods require a significant ANOVA, MANOVA, or Tukey's range test. These methods generally provide only "weak" control of Type I error, except for certain numbers of hypotheses." But I think I get your point. Thanks! $\endgroup$ – Paul Aug 1 '19 at 13:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.