Doing multiple comparisons increases the number of false positive findings. In the Bonferroni method, the adjusted critical level is $\hat{\alpha} = \frac{\large\alpha}{N}$ where $N$ is the number of tests. However, demanding this from each test increases the number of false negatives.

In the Holm-Bonferroni and FDR methods, the smallest p-values have to be likewise below $\frac{\large\alpha}{N}$. Is this some general property?

In order to control the number of false positives, when doing $N$ tests, do the smallest $p$ have to be smaller than $\alpha/N$? It seems the Holm-Bonferroni and FDR differ only with respect to the following values.


Simply put, no, it is not a general property of familywise error rate (FWER) control or of false discovery rate (FDR) control that the smallest p-value must be smaller than α/N.

The Holm-Bonferroni procedure is a "step-down" method, so the significance of each p-value depends on the lowest p-value. However, that is not the case when using "step-up" methods, such as the Hochberg procedure (for controlling the FWER) and the Benjamini-Hochberg procedure (for controlling the FDR).

For example, say you conduct 3 tests and get the following p-values: .02, .03, and .04. None of those p-values are < α/N (assuming α=.05), yet all of them would be significant using a step-up procedure such as the ones I mentioned.

  • $\begingroup$ Other exceptions (just to mention a FWER controlling one) would be Dunnett's test (and related test such as the one by Dunnett and Tamahane), where the threshold is a little bit higher than $\alpha/N$, weighted Bonferroni(-Holm) tests, various closed testing procedures etc. Just in case the list above may have given the impression that that the smallest p-value must be smaller than $\alpha/N$ for FWER controlling methods. $\endgroup$
    – Björn
    Mar 1 '17 at 13:13
  • $\begingroup$ As I noted, the Hochberg method (not to be confused with the Benjamini-Hochberg method) is for controlling the FWER. But you are right that there are numerous other FWER-controlling methods that don't require p < α/N for the lowest p-value. Dunnett's is a good example. $\endgroup$
    – Bonferroni
    Mar 1 '17 at 21:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.