I'm finding an issue while adjusting p-values with Holm's method I cannot understand.

It's not a problem in my actual data, but I would appreciate any help to understand the issue.

To show the question I have created some p-values from 0.04 to very low values, then compute bonferoni's and Holm's adjusted p-values

myp <- data.frame("unadjusted" = 0.04 / c(1, 5, 10, 50, 100, 500, 1000)) # dummy p-values
myp$bonferroni <- p.adjust(myp$unadjusted, method = "bonferroni") # Bonferroni's method
myp$holm <- p.adjust(myp$unadjusted, method = "holm") # Holm's one
myp$holmreject0.05 <- ifelse(myp$holm < 0.05, "yes", "no") # test if adjusted p-value is below alpha threshold
format(myp, scientific = FALSE)

  unadjusted bonferroni    holm holmreject0.05
1    0.04000    0.28000 0.04000            yes   
2    0.00800    0.05600 0.01600            yes
3    0.00400    0.02800 0.01200            yes
4    0.00080    0.00560 0.00320            yes
5    0.00040    0.00280 0.00200            yes
6    0.00008    0.00056 0.00048            yes
7    0.00004    0.00028 0.00028            yes

Bonferroni is easy to understand: 7 comparisons, p-value multiplied by 7. About Holm I have got quite surprised noticing the first p-value wasn't modified, reading the Wikipedia I understand that Holm's method more strongly modifies the first (lowest) p-value, and then goes ahead till the last (highest) one, which is "less penalized" (step-down method). With this dummy p-values, they have been multiplied by 7 (the lowest) to 1 (the highest).

In this question, focused at a different question, an user posted a table, and their higher p value was not modified either (sixth file). I was also multiplied by 1.

Going further, if I go to Wikipedia example on the method and use their data, the highest p-value is in fact modified:

WikipediaExample <- data.frame("unadjusted" = c(0.01, 0.04, 0.03, 0.005)) # unadjusted p-values
WikipediaExample$adjusted <- p.adjust(WikipediaExample$unadjusted, method = "holm") # Holm's adjusted
WikipediaExample$reject0.05 <- ifelse(WikipediaExample$adjusted < 0.05, "yes", "no") # over/below alpha threshold 0.05

  unadjusted adjusted reject0.05
1      0.010     0.03        yes
2      0.040     0.06         no  # highest p-value, multiplied by 1.5
3      0.030     0.06         no
4      0.005     0.02        yes

I cannot understand how is it possible that in Wikipedia example, a p = 0.04 is over 0.05 threshold with 4 comparisons, but in the dummy data I created, with 7 comparisons a p = 0.04 is below 0.05 threshold. I expected the more comparisons, the more penalized p...

Thanks in advance.

Note: my maths level (I'm a physician) is not enough to read the formula for adjusting p-values. I can understand and check with pencil and paper the first part of Wikipedia entry on rationale and formulation


One way to think of this: Suppose you have a bunch of tests that all have $p<0.05$. The multiple comparisons make it easier to get one $p$-value less than $0.05$, but make it harder to get all the $p$-values less than 0.05. No matter what the correlation between these tests, that's going to happen less than 5% of the time under the null.

So, the largest $p$-value should be adjusted to less than 0.05, and in fact ends up not changing.

  • $\begingroup$ I think I get the point: if we have a lot of comparisons, the last ones (the largest) will be less impacted (divided by a number closer to 1). In the wikipedia example, as few comparisons were made the last ones are still quite impacted. $\endgroup$ – Miguel Menéndez Jun 5 '20 at 14:17
  • 1
    $\begingroup$ Yes, exactly. The largest one isn't (obviously!) being selected for being small, so it doesn't have to be penalised for selection. $\endgroup$ – Thomas Lumley Jun 5 '20 at 22:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.