2
$\begingroup$

In section 2 of this PDF, it is stated that the Bonferroni correction "benefits" from the tests being independent, and that if tests are not independent, the Bonferroni correction could be far too conservative.

The Bonferroni correction says to use a significance level of $\alpha / n$ when conducting $n$ tests. It is stated that the probability of at least one false positive (null hypothesis is true but we reject it) at a significance level of 5% with 20 tests using the Bonferroni correction is

$$ P(\text{at least 1 significant result}) = 1 - P(\text{no significant results}) $$ $$ 1 - (1 - 0.0025)^{20} = 0.0488 $$

It is not difficult to see that this is true for 20 independent tests. But it is claimed that dependent tests can cause the test to be extremely conservative. How? Does $1-P(\text{no significant results})$ reach its maximal value for independent tests?

Why is that so? https://www.stat.berkeley.edu/~mgoldman/Section0402.pdf

$\endgroup$
1
  • 1
    $\begingroup$ Run the same test on the same data 20 times. Apply the Bonferroni correction: what does it do to the p-value? Draw the conclusion. $\endgroup$
    – whuber
    Commented Jul 3, 2020 at 16:06

1 Answer 1

3
$\begingroup$

I think people fixate too much on the result under independence, and miss sight of where Bonferonni correction is fundamentally coming from. A basic probability axiom (or can be obtained as a result of a measure-theoretic approach) is that for any events $A,B$, $$P(A\cup B) = P(A) + P(B) - P(A \cap B)$$

and since probabilities are weakly positive, we have that $P(A\cup B) \leq P(A) + P(B)$. More generally, we have that for any set of events $\{E_k\}_{k=1}^K$, $$P(\cup_{k=1}^K E_k) \leq \sum_{k=1}^K P(E_k)$$

and this bound holds with equality when the events are all disjoint. Let each $k$-th event be the event of having a significant result for the $k$-th test. So Bonferonni is just considering the worst case scenario. This worst case scenario is actually not fully attained when the tests are independent (disjoint and independent are different concepts), and the Bonferonni result fundamentally comes from this mathematical upper bound on the probability of the union of events. To see this, simply note that Bonferroni tells us to reject the null for each $k$ where $p_k \leq \frac{\alpha}{K}$. Suppose $K_0$ of these $K$ tests are actually null. Then the probability of rejecting at least one true null hypothesis follows from considering the probability of rejecting any of the $K_0$ tests. We thus have

$$P(\cup_{k=1}^{K_0} E_k) \leq \sum_{k=1}^{K_0} P(E_k) = \sum_{k=1}^{K_0} P(p_k \leq \frac{\alpha}{K}) = K_0\frac{\alpha}{K} \leq K \frac{\alpha}{K} = \alpha$$

as required.

Again, the key is that this follows from basic laws of probability and the best bound we can place on the probability of a union of events. The case of independent tests is simply one setting that is often illustrated, but the result fundamentally relates to disjoint events and this axiom of probability. As you mention in your comment, with 20 tests (or pretty much any number of tests), if they are all indeed independent, then the probability of rejecting a null test is about $.487$ or so, as you calculated. Looking back to the simple case, the reason is that $P(A\cap B) = P(A)P(B)$ when independent, whereas it is $P(A\cap B) = 0$ when disjoint. So when independent, you actually subtract a bit from the sum of individual events, hence the value is actually a bit smaller than $\alpha$ (so even for independent tests, Bonferonni is a bit too conservative, but it's very close!!).

$\endgroup$
3
  • 1
    $\begingroup$ The Bonferroni correction is also much less conservative than many people think, especially when $K$ is large. The conservatism of the correction gets confounded with the much more severe conservatism of controlling family-wise error rate. $\endgroup$ Commented Jul 3, 2020 at 20:27
  • 1
    $\begingroup$ Thanks for your answer. One thing I don't understand is why the worst case scenario occurs with independent tests. If you do the calculation for independent tests, like I did in the question you get 0.0488. This is less than the worst case (0.05), which is achieved for disjoint tests. $\endgroup$
    – Sahand
    Commented Jul 4, 2020 at 14:13
  • 1
    $\begingroup$ @Sahand great comment. I mis wrote what I meant to share.. updated my post! As mentioned, the bound is actually too conservative for even independent, and it really binds only for disjoint events, and independent events need not be disjoint. But it's quite close to the bounds under independent (as you see with your calculcations), but yes, technically, even under independent, Bonferonni is too conservative (but just slightly so!) $\endgroup$
    – doubled
    Commented Jul 4, 2020 at 17:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.