When using a Holm-Bonferroni adjustment method or the like, how to you treat multiple p-values of the same value when ranking?

For instance, I have the unadjusted p-values:


Should both 2.20E-16 values be treated as rank (1) and if so, should the 0.001387 value be rank (2) or (3)? Or should one of the 2.20E-16 values be (1) and the other (2), which would make the 0.001387 value (3)?


  • $\begingroup$ Ignoring the impact of the higher values for the present, the third one should definitely be rank 3 and the first two shouldn't both be 1. Keeping in mind that 2.2E-16 is simply the smallest p-value R will print, they're almost certainly different and if you know the test statistics for the first two you can probably rank their p-values without knowing the exact p-values (but you probably don't need to know which is which, just that one is 1 and one is 2) $\endgroup$ – Glen_b -Reinstate Monica Oct 11 '19 at 6:22

The only difference it makes is in terms of the adjusted p-values, not in terms of whether the tests are significant. I have never seen a proposal for exactly how to do this and it probably does not matter too much (never mind getting too obsessed about statistical significance - getting into comparing the size of "significant" p-values is certainly completely pointless).

If you are in a situation, where two p-values are exactly the same and the associated null hypotheses are getting rejected, then I would be tempted to go with the option of adjusting each one as if it had been rejected before the other. This is in a sense the "conservative" approach in that it results in higher p-values compared to, say, randomly picking one of them to have been rejected first. Doing so does not change anything else for any other tests.


In the Holm approach, you immediately stop as soon a p value is above its treshold. So it does not matter.

Since Holm-correction is just for decision making and not for creating p values, in your situation nothing changes anyway if you flip the order of the first.

Here are two more interesting situations.

Situation 1

P values are 0.02, 0.02, 0.03

The smallest p value times 3 is above alpha 0.05. So the algo stops and all tests are non-significant. Same if you switch identical p values.

Situation 2

P values are 0.01, 0.01, 0.03

The smallest p value times 3 is below 0.05. So the algo continues with the next smallest p value times 2 (okay) etc. All tests are significant. Same if you switch identical p values.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.