I have a bit of trouble understanding the Holm-Bonferroni correction. For the normal Bonferroni correction you simply divide your alpha level by the number of tests. For example, if I have 3 tests, I will test all 3 test against a alpha level of .016. Note: I´m adjusting alpha instead of p, because I´m doing a power analysis.

Now, for the Holm-Bonferroni correction, I understood that you apply the Bonferroni correction sequentially. That is: Test 1: Alpha = .05. Test 2: Alpha = .025 Test 3: Alpha = .016. However, that seems to be incorrect. Consider the following code in R:

> pvalues <- c(0.049, 0.049, 0.049)
> p.adjust(pvalues, method = "holm")
[1] 0.147 0.147 0.147
> p.adjust(pvalues, method = "bonferroni")
[1] 0.147 0.147 0.147

Both methods return exactly the same output for all three p-values. Can anyone explain where I get the Holm-Bonferroni correction wrong and how I can adjust my alpha appropriately? Please note that the output remains identical with more p-values.

Thank you!


1 Answer 1


Maybe I'm getting it wrong, but it seems that both methods return the same corrected p-values because the three original p-values are equal.

For the Bonferroni correction, you simply multiply each p-value by the number of p-values (here by $3$).

For the Holm-Bonferroni, first you need to sort the p-values and then multiply the smallest by $3$, then the second one by $2$ etc.
But if at one step the corrected p-value is smaller than the previous one, then it is made equal to it (this sentence is only based on empirical constatations using the p.adjust function).

For example, say your (sorted) p-values are $(0.1,0.11,0.5)$.
The first one is multiply by $3$: $0.1 \leftarrow 0.3$, the second one by $2$: $0.11 \leftarrow 0.22$.
But since $0.22 < 0.3$, the corrected p-value for the second one will be $0.3$. For the third you multiply by $1$. The corrected p-values are then $(0.3,0.3,0.5)$.

I think this is what happens here, since all the p-values are equal you have the first one multiplied by $3$ which gives $0.147$, the second one by $2$ which gives $0.098$. Since $0.098 < 0.147$, actually the corrected p-value for the second one will be $0.147$. The same goes for the third one.

Then all the three corrected p-values will be the same (and equal to those given by the Bonferroni correction since the have been multiplied by $3$).

If you use different p-values, the Holm method should give you something different than the Bonferroni method.

  • $\begingroup$ Thank you for your reply. I think I understood it now. However, your last sentence is incorrect: It will give you the same p-value if the differences are small. > pvalues <- c(0.048, 0.047, 0.049) > p.adjust(pvalues, method = "holm") [1] 0.141 0.141 0.141 > p.adjust(pvalues, method = "bonferroni") [1] 0.144 0.141 0.147 > $\endgroup$
    – Max
    Jul 12, 2020 at 8:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.