0
$\begingroup$

I conducted a one-way ANOVA: I had two groups and tested, if they differ in three different DVs.

Since I tested three times, I thought it was wise to go for Bonferroni-correction. But then I got the notification that the correction can't be done because there are less than three groups. Can someone explain this to me? I thought the Bonferroni-correction was exactly for this case: testing data on the same groups several times.

$\endgroup$
4
  • $\begingroup$ One-way ANOVA procedure does not know that you treat these 3 individual analyses as jointly belonging to one family of tests. The ANOVA thinks instead, that you want to do the multiple comparisons between groups, and since you have only 2 groups, the correction is not applied. $\endgroup$
    – ttnphns
    May 24 '14 at 8:42
  • $\begingroup$ @ttnphns: Thank you! So if I do the correction manually now, is the new alpha-level also valid in case I add a control variable through ANCOVA? $\endgroup$
    – Jennifer
    May 24 '14 at 11:46
  • $\begingroup$ Sure. But I doubt that you need to control for "multiple analyses" at all. You may, though. You may even add all those tests (to the list of multiple analyses) that you did last year :-) $\endgroup$
    – ttnphns
    May 24 '14 at 12:11
  • $\begingroup$ @ttnphns While family-wise error rate corrections change with a redefinition of "family", false discovery rate inferences do not. $\endgroup$
    – Alexis
    May 25 '14 at 13:44
1
$\begingroup$

It appears that instead of conducting a one-way ANOVA, you want to conduct a one-way MANOVA, since you are comparing groups on multiple DVs. In this situation, there's no reason Bonferroni correction shouldn't be done for the three comparisons. Just divide your alpha level by 3 for each of the 3 tests. You don't even need to conduct the MANOVA omnibus test.

Another approach, which is valid but rarely used, is to first conduct the MANOVA omnibus test and then divide your alpha by 3-1=2 (i.e. divide by the number of tests minus 1) for the individual tests--but forfeit significance for all 3 tests if the omnibus test isn't significant. In 2-group designs, this approach controls the Type I error rate for any number of DVs (e.g., for 10 DVS, you can use alpha/9 for the individual tests), provided that the significance of all tests is conditional on significance of the omnibus test. [Reference: Frane, A.V., 2015, "Power and Type I error control for univariate comparisons in multivariate two-group designs" Multivariate Behavioral Research, 50]

If you were instead comparing 3 groups on a single DV, then you could still do a Bonferroni correction for the 3 comparisons. Again, you could do this without conducting an omnibus test. Another approach is to first conduct an ANOVA omnibus test and then conduct the individual tests at the unadjusted alpha level--but forfeit significance for all 3 tests if the omnibus test isn't significant. Note that for more than 3 groups, using the unadjusted alpha level wouldn't control the familywise error rate, so this is a special case where you can get by without adjusting.

In short, you can ALWAYS use Bonferroni to control the error rate for multiple tests, so no person (or software) should tell you that you can't. But in many situations there are other options that use smaller reductions (or no reduction at all) in the comparisonwise alpha level.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.