17
$\begingroup$

Reading Field's Discovering Statistics Using SPSS (3rd Edition) I was struck by a bit about post-hoc tests in ANOVA. For those wanting to control the Type I error rate he suggests Bonferroni or Tukey and says (p. 374):

Bonferroni has more power when the number of comparisons is small, whereas Tukey is more powerful when testing large numbers of means.

Where should the line be drawn between a small and large number of means?

$\endgroup$
  • 5
    $\begingroup$ At the bottom of the following Web page from NIST/Sematech, itl.nist.gov/div898/handbook/prc/section4/prc473.htm,it is recommended that both tests should be preformed and the smaller of the two intervals be taken. I have found similar comments in Johnson and Wichern on doing MANOVA. $\endgroup$ – schenectady Apr 11 '11 at 12:31
  • 1
    $\begingroup$ @schenectady Good answer! Why don't you paste it into a reply? BTW, the link is corrupted in your comment; the correct one is itl.nist.gov/div898/handbook/prc/section4/prc473.htm . $\endgroup$ – whuber Apr 11 '11 at 15:07
  • 1
    $\begingroup$ First one brief point: Power is directly related to the Type II error rate, not the Type I. Now forgive me but I'm going to spout off some opinions. Might what you're doing be seen as "gaming the system," trying to rig it so that more results get classified as sig. or nonsig.? These binary judgments are so much less informative, and potentially so much more misleading, than reports of actual effect sizes--in your case, regarding group differences in means. I like to see people use p-values to garnish results rather than to structure them. End of editorial--argue away! $\endgroup$ – rolando2 May 11 '11 at 22:32
  • 1
    $\begingroup$ "At the bottom of the following Web page from NIST/Sematech, itl.nist.gov/div898/handbook/prc/section4/prc473.htm, it is recommended that both tests should be preformed and the smaller of the two intervals be taken. I have found similar comments in Johnson and Wichern on doing MANOVA. – @schenectady Apr 11 '11 at 12:31" This is considered data mining and shouldn't be done. The choice of tukey vs. Bonferroni should be made prior to analysis. $\endgroup$ – user16992 Nov 19 '12 at 16:18
  • $\begingroup$ Minitab's online documentation appears to offer similar advice support.minitab.com/en-us/minitab/17/topic-library/… $\endgroup$ – N Brouwer Mar 14 '17 at 19:17
5
$\begingroup$

In addition to the useful link mentioned in the comments by @schenectady.

I would also add the point that Bonferroni correction applies to a broader class of problems. As far as I'm aware Tukey's HSD is only applied to situations where you want to examine all possible pairwise comparisons, whereas Bonferroni correction can be applied to any set of hypothesis tests.

In particular, Bonferroni correction is useful when you have a small set of planned comparisons, and you want to control the family-wise Type I error rate. This also permits compound comparisons. For example, you have a 6-way ANOVA and you want to compare the average of groups 1, 2, and 3 with group 4, and you want to compare group 5 with 6.

To further illustrate, you could apply Bonferroni correction to assessing significance of correlations in a correlation matrix, or the set of main and interaction effects in an ANOVA. However, such a correction is typically not applied, presumably for the reason that the reduction in Type I error rate results in an unacceptable reduction in power.

$\endgroup$
  • $\begingroup$ just curious if you have a citations for: "However, such a correction is typically not applied, presumably for the reason that the reduction in Type I error rate results in an unacceptable reduction in power." Thanks a ton! $\endgroup$ – user16970 Nov 18 '12 at 19:53
  • $\begingroup$ Welcome to the site. This should be posted as a comment, not an answer. $\endgroup$ – Peter Flom Nov 18 '12 at 21:49
  • $\begingroup$ @Jessica. No I don't have a citation for that claim. But it is quite easy to show through simulation, formulas, or even just a basic knowledge of the factors that affect statistical power (i.e., such factors include alpha). $\endgroup$ – Jeromy Anglim Nov 19 '12 at 2:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.