For dealing with multiple testing, Benjamini and Hochberg (1995) assume that the tests are independent. Benjamini and Yekutieli (2001) show that (1) the 1995 result also holds under (I think) weaker conditions and that (2) including the factor $\Sigma^m_{i=1}\frac{1}{i}$ makes the procedure applicable for any dependence structure (see Theorem 1.3). Since this summation is greater than or equal to 1, it would make the procedure more conservative.

My question is why is independent tests not the worst case assumption? Is there some simple example to illustrate why not? Does it involve a negative dependence structure of some sort?

Maybe my understanding is faulty, but I thought that the Bonferroni adjustment (for the familywise error) worked for any dependence and that independence is the worst case assumption. Perhaps this is not correct and hence my confusion with BH.


Independence is more like a best-case assumption than a worst-case assumption. Loosely, when data are independent, each datum contains as much information as possible. If data were dependent, because their values can be predicted from other data, each additional datum must have less new information to contribute (the part that could have been predicted you already knew in some sense). The situation can be similar with multiple testing. In terms of simple alpha correction strategies, if the tests are independent, the Dunn-Sidak correction can be used:
$$ \alpha_{\rm DS} = 1 - (1-\alpha)^{1/k} $$ but if the tests are not independent, the Bonferroni correction must be used:
$$ \alpha_{\rm B} = \frac{\alpha}{k} $$ As is clear from the formulas, $\alpha_{\rm DS}\ge \alpha_{\rm B}$.

  • 1
    $\begingroup$ gung I would love it if you could frame your answer in terms of the false discovery rate presented by Benjamini and Hochberg and later by Benjamini and Yeuketeli, rather than in terms of the familywise error rate. $\endgroup$ – Alexis Aug 19 '14 at 19:18
  • $\begingroup$ @gung I see what you are saying about Dunn-Sidak versus Bonferroni. But I still think independence is (close to) the worst case situation for Bonferroni. While admittedly extreme, perfect correlation in my mind is the best case for Bonferroni in that I could use $\alpha$ with no adjustment. Are we using "best" and "worst" in the same way? I mean best in the sense of requiring the smallest downward adjustment of the p-value (not sure if this is standard usage). Thanks. $\endgroup$ – kkoro Aug 19 '14 at 20:03
  • 1
    $\begingroup$ @kkoro, if the tests are independent, you use DS, & if they are dependent you use B. There is no independence w/ B. Since the B correction is more extreme than the DS correction, that shows that independence requires a lesser correction. This is just a quick intuition pump. $\endgroup$ – gung - Reinstate Monica Aug 19 '14 at 20:12

Gung is incorrect that Dunn-Sidak should be used under independence and Bonferroni must be used under dependence. In fact, Dunn-Sidak controls the FWER not only under independence, but also under positive dependence. And Bonferroni controls the FWER for any dependence structure--including independence.

To answer your question, the reason independence isn't the "worst case" is that in certain situations, you can theoretically have negative dependence.

  • 1
    $\begingroup$ Where did @gung deny that "Bonferroni works for any dependence structure"? $\endgroup$ – amoeba says Reinstate Monica Jul 19 '16 at 15:39
  • $\begingroup$ Gung didn't deny that, but did imply that Sidak is only valid under independence: "if the tests are independent, the Dunn-Sidak correction can be used...but if the tests are not independent, the Bonferroni correction must be used." $\endgroup$ – Bonferroni Jul 19 '16 at 19:19
  • $\begingroup$ I would suggest that you try to be very specific when you say that somebody "is incorrect". $\endgroup$ – amoeba says Reinstate Monica Jul 19 '16 at 21:43
  • $\begingroup$ Great suggestion! Gung claimed that Dunn-Sidak should be used under independence and Bonferroni must be used under dependence. As I stated, quite specifically, that is incorrect. For controlling the FWER, Bonferroni can be used under independence or under any kind of dependence, and Dunn-Sidak can be used as long as there is not negative dependence. I hope that is clear and specific enough. $\endgroup$ – Bonferroni Jul 19 '16 at 22:32
  • $\begingroup$ Thanks, but I already understood what you meant from your previous comment. My suggestion was that you edit your answer to be clearer :) $\endgroup$ – amoeba says Reinstate Monica Jul 19 '16 at 22:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.