Bonferroni Correction: Independence vs Disjoint Events

Question

Let $T_i$ and $c_i$ be a test statistic and critical value, respectively, for testing hypothesis $H_i$, $i = 1,2,\dots, n$. Assume a Type-1 error rate of $\alpha$ for each test. The family-wise error rate is given by

$$\textit{FWER} :=P\left(\bigcup_{i=1}^n\{T_i>c_i\}\right).$$

If the events $\{T_i\leq c_i\}$, $i=1,2,\dots,n$ are independent, the $\textit{FWER}$ can be rewritten as $$P\left(\bigcup_{i=1}^n\{T_i>c_i\}\right) = 1-P\left(\bigcap_{i=1}^n\{T_i\leq c_i\}\right) = 1 - \prod_{i=1}^nP(\{T_i\leq c_i\} = 1 - (1-\alpha)^n.$$

The Bonferroni method replaces $\alpha$ with $\alpha^* := \frac\alpha n$ to ensure that $\textit{FWER} \leq \alpha$. In fact, using the inequality $1-\exp(-x)\leq x$ for all $x\in\mathbb R$, we can show that $\textit{FWER}\leq\alpha$ if we chose a Type-1 error rate of $\alpha^*$ for each test.

The Bonferroni correction is a consequence of the sub-additivity property of probability measures:

$$P\left(\bigcup_{i=1}^n\{T_i>c\}\right) \leq \sum_{i=1}^n P(\{T_i>c_i\} = n\alpha.$$

Now I was wondering when is the bound attained? By the additivity property of probability measures for disjoint sets, this is the case if the events $\{T_i>c_i\}$, $i=1,2,\dots,n$ are mutually disjoint.

Since the events $\{T_i\leq c_i\}$ cannot be independent while the events $\{T_i>c_i\}$ are disjoint, and vice versa (see my other question on MSE https://math.stackexchange.com/questions/4723024/subadditivity-of-probability-measure-and-independence), it seems to be true that independence is not as worse as disjointness of events with regard to Type-1 error rate cumulation. Is this conclusion correct? Because I have often read that independence of hypotheses is the worst case.

I suppose that hypotheses concerning different data sets are independent. But what would be an example of disjoint events?

Answer 1

You are right. Independence is not the worst case. The argument is as stated by you. The FWER in case of independence is easy to compute and normally not far away from the worst case, but it isn't the worst case itself, as stated. Different data sets are not necessarily independent (although usually taken as independent), however the kind of dependence that leads to the worst case is hard to achieve with different data sets in a real situation.

A very simple example for the worst case is standard testing of a parametric hypothesis $H_0:\ \mu=0$ on the same data separately against both $\mu<0$ and $\mu>0$. The rejection events will be disjoint in most cases (think of a t-test of a mean). They're obviously not independent, because if you reject one, you will not reject the other. If you run both of these tests at level $\alpha/2$ as prescribed by Bonferroni, putting the tests together gives the usual two-sided test against $\mu\neq 0$ at level $\alpha$.