2
$\begingroup$

A colleague was using three pair-wise t-tests plus Bonferroni correction to detect any differences in the means of three groups. My first response was to tell him to use an ANOVA instead, since my understanding is that's the "correct" way to detect a difference in means between 3 or more groups.

However, if multiple comparisons are being controlled for, as they were in his case, is the ANOVA still best? Does it depend on the context?

(To test, I made a quick simulation of 3 distributions of uniformly distributed random numbers, applying a variable offset to the mean of one of the groups. Using a Bonferroni correction the two methods were almost always in agreement (98% of the time for the offsets I used).)

Improve this question
$\endgroup$
1
  • $\begingroup$ Have a look here and here. Does this help answering your question? $\endgroup$
    – Stefan
    Jan 20 '16 at 1:40
2
$\begingroup$

ANOVA will simultaneously test if all means are equal in a single test. If the test fails to reject the null hypothesis, then you can stop with evidence that there is no difference in means. However, if there are differences (i.g. you reject) you'd still need to perform multiple testing to determine which pairs of means differ from each other.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks. I understand that's the way it should be done, and that's the way I would do it. But I struggled to justify it to him. Why is starting with multiple tests always worse than starting with ANOVA if the multiple comparisons are being controlled for? $\endgroup$
    – R Greg Stacey
    Jan 20 '16 at 1:09
  • $\begingroup$ Well, you don't HAVE to run ANOVA first -- this is just how multiple comparisons are presented in textbooks and multiple comparison in and of themselves have nothing to do directly with ANOVA. Multiple comparisons using Bonferroni adjustments are too conservative and a simple ANOVA is more powerful for detecting a single difference among the groups -- especially with a lot of comparisons. So, if you are really just interested in knowing if any pair (regardless of which pair) of groups is different the ANOVA is the test that you should use because of it's power. $\endgroup$
    – StatsStudent
    Jan 20 '16 at 1:57
  • 1
    $\begingroup$ Lastly, if your college plans to use multiple comparison tests, there are usually better test to employ (read more powerful) than the Bonferroni adjusted tests depending on the comparisons you are interested in. For example, if you plan to perform all possible comparisons, Tukey's HSD test preferred to the Bonferroni adjustement. This page provides a nice overview of the advantages and disadvantages of the tests for different comparison conditions: www2.hawaii.edu/~taylor/z631/multcomp.pdf $\endgroup$
    – StatsStudent
    Jan 20 '16 at 2:03
  • $\begingroup$ @StatsStudent the link is broken $\endgroup$
    – Erik Hambardzumyan
    Sep 16 '20 at 10:16
  • $\begingroup$ @ErikHambardzumyan Try this (thanks to the internet archive): web.archive.org/web/20181024155101/http://www2.hawaii.edu:80/… $\endgroup$
    – StatsStudent
    Sep 17 '20 at 5:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.