6
$\begingroup$

OK, so I have absolutely scoured the internet, and it appears that there is consensus (among people who really care about Type I error) that doing a MANOVA before doing multiple ANOVAs does nothing to protect against Type I error. But then, there's still a set of people I see that are saying that MANOVA DOES protect against Type I error, and that if the MANOVA is significant, you can simply run ANOVAs and t-tests with p<.05 without multiple comparison correction.

For example, this video here: https://www.youtube.com/watch?v=3pzCa4Whv74

They do MANOVA -> multiple ANOVAs -> multiple t-tests. There is no multiple comparison (e.g., Bonferroni, etc. etc.). The fact is that the MANOVA (if significant) justifies the multiple ANOVAs (which you can then use a significance threshold of .05), which then (if significant) justifies the t-tests.

My question is two-fold: a) is the practice above still considered "reasonable"? For example, if I submitted a paper in Computer Science using the above process, would I get outright rejected (e.g., with ~50 dependent variables). and b) are the majority of the people doing MANOVAs in published papers still doing the above? (I imagine the answer is yes)

$\endgroup$
3
$\begingroup$

Requiring a significant MANOVA or ANOVA before conducting individual t-tests can help protect against Type I error, but is not generally sufficient to provide strong Type I error control. For example, if you only conduct the individual tests when the MANOVA is significant, that controls the familywise Type I error rate when the all null hypotheses are true (this is sometimes called "weak" control). But that doesn't necessarily protect you when one or more null hypotheses are false (unless there are only 2 DVs, in which case one null hypothesis being false would eliminate the problem of multiple comparisons). Similarly, if you only conduct the individual tests when the ANOVA is significant, that controls the familywise Type I error rate when all group means are equal, but not when one or more group means are different from the rest (unless there are only 3 groups in which case one group mean being different would eliminate the problem of multiple comparisons).

Therefore, the method you described does not reliably control Type I error. However, you can use a similar method if you apply a multiple comparisons adjustment (see references below).

References:

Frane. 2015. Power and Type I error control for univariate comparisons in multivariate two-group designs. Multivariate Behavioral Research, 50.

Hayter, 1986. The maximum familywise error rate of Fisher's least significant difference test. Journal of the American Statitical Association, 81.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.