How does the Bonferroni correction benefit from independent tests?

Question

In section 2 of this PDF, it is stated that the Bonferroni correction "benefits" from the tests being independent, and that if tests are not independent, the Bonferroni correction could be far too conservative.

The Bonferroni correction says to use a significance level of $\alpha / n$ when conducting $n$ tests. It is stated that the probability of at least one false positive (null hypothesis is true but we reject it) at a significance level of 5% with 20 tests using the Bonferroni correction is

$$ P(\text{at least 1 significant result}) = 1 - P(\text{no significant results}) $$ $$ 1 - (1 - 0.0025)^{20} = 0.0488 $$

It is not difficult to see that this is true for 20 independent tests. But it is claimed that dependent tests can cause the test to be extremely conservative. How? Does $1-P(\text{no significant results})$ reach its maximal value for independent tests?

Why is that so? https://www.stat.berkeley.edu/~mgoldman/Section0402.pdf

Answer 1

I think people fixate too much on the result under independence, and miss sight of where Bonferonni correction is fundamentally coming from. A basic probability axiom (or can be obtained as a result of a measure-theoretic approach) is that for any events $A,B$, $$P(A\cup B) = P(A) + P(B) - P(A \cap B)$$

and since probabilities are weakly positive, we have that $P(A\cup B) \leq P(A) + P(B)$. More generally, we have that for any set of events $\{E_k\}_{k=1}^K$, $$P(\cup_{k=1}^K E_k) \leq \sum_{k=1}^K P(E_k)$$

and this bound holds with equality when the events are all disjoint. Let each $k$-th event be the event of having a significant result for the $k$-th test. So Bonferonni is just considering the worst case scenario. This worst case scenario is actually not fully attained when the tests are independent (disjoint and independent are different concepts), and the Bonferonni result fundamentally comes from this mathematical upper bound on the probability of the union of events. To see this, simply note that Bonferroni tells us to reject the null for each $k$ where $p_k \leq \frac{\alpha}{K}$. Suppose $K_0$ of these $K$ tests are actually null. Then the probability of rejecting at least one true null hypothesis follows from considering the probability of rejecting any of the $K_0$ tests. We thus have

$$P(\cup_{k=1}^{K_0} E_k) \leq \sum_{k=1}^{K_0} P(E_k) = \sum_{k=1}^{K_0} P(p_k \leq \frac{\alpha}{K}) = K_0\frac{\alpha}{K} \leq K \frac{\alpha}{K} = \alpha$$

as required.

Again, the key is that this follows from basic laws of probability and the best bound we can place on the probability of a union of events. The case of independent tests is simply one setting that is often illustrated, but the result fundamentally relates to disjoint events and this axiom of probability. As you mention in your comment, with 20 tests (or pretty much any number of tests), if they are all indeed independent, then the probability of rejecting a null test is about $.487$ or so, as you calculated. Looking back to the simple case, the reason is that $P(A\cap B) = P(A)P(B)$ when independent, whereas it is $P(A\cap B) = 0$ when disjoint. So when independent, you actually subtract a bit from the sum of individual events, hence the value is actually a bit smaller than $\alpha$ (so even for independent tests, Bonferonni is a bit too conservative, but it's very close!!).

Answer 2

Imagine the highly dependent case when there are 2 tests on the same data, and one is a copy of the other, per bonferroni correction, we can only use \alhpa/2 to test for significance for either of these 2 tests, but actually there is only one test tested. Like if someone is pretty, do I need to test/guess how his/her twin brother/sister look like? probably not, i will just pick either one and test it without correction of alpha.