I already read the answers here: About the Bonferroni correction, but I still don't understand the concept completely.

Assume the following hypothetical case, where I perform an ANOVA test to evaluate if 4 means are equal. As this test turns out significant (p < .05, i.e. at the 5% level), I want to do pairwise tests to see if some means may be equal. Because I have 4 means to be tested, that's 6 extra tests I should do.

My question: should these pairwise tests have a p-value of smaller than .05/6 = 0.0083 to be significant? Or is a correction of 0.05/2 for each follow up test sufficient?


Each of $k$ tests has its own potential false positive with a chance of $\alpha$, so you could reason that the combined chance of at least one false positive is:

$$1 - (1 - \alpha)^k$$

With the Bonferroni correction we try to control this family-wise error rate (FWER) by multiplying each $p$-value by the number of tests you perform ($k=6$ in your case). The new $p$-values are then:

$$p_{\text{Bonferroni}} = p_{\text{original}} \cdot k$$

Equivalently, you can divide the level of significance $\alpha$ by $k$ and only consider $p$-values below this new threshold to be significant.

You can see for yourself that this works, for example by setting $\alpha = 0.05$ and $k = 10$:

$$1 - \Big(1 - \frac{\alpha}{k}\Big)^k = 1 - (1 - 0.005)^{10} \approx 0.05$$

Which is the desired FWER.

If this correction is too severe, consider the following:

  1. Do you really need every individual comparison?
  2. Can't you use the standard approach to post-hoc analysis after ANOVA (Tukey's HSD)?
  3. Have you considered a different correction (e.g. Benjamini-Hochberg)?

If you are comparing methods to some baseline for example, you could opt not to compare every individual category, which will reduce $k$.

Rather than applying a Bonferroni correction, it is more common to perform post-hoc comparisons of means in ANOVA using Tukey's honest significant difference test. This test already corrects for multiple testing.

Lastly, if for some reason you do not want to use Tukey's HSD, and still find Bonferroni to be too conservative, you could use a different correction. For example, the Benjamini-Hochberg procedure controls the false discovery rate instead of the FWER. The idea behind this is roughly as follows:

  • If the most significant result is still significant after the Bonferroni penalty, then perhaps not every test has its own false positive (since we found at least one significant effect after correction). Hence, the Bonferroni correction might be too harsh and we correct the second most significant $p$-value by a less severe penalty.
  • 1
    $\begingroup$ It is not usual however to use Benjamini Hochberg in this case (it is used mostly for high dimensional datasets, where you test your hypothesis hundreds or thousands of times, e.g. on many genes). Holm is less conservative than Bonferroni and probably better suited. Which one you should use depends on how conservative you want to be which in turn depends on the aim of the experiment. Do false positive findings matter much? With how much certainty do you want to be able to draw your conclusions? $\endgroup$ – jkd Aug 27 '18 at 17:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.