Is Bonferroni correction too anti-conservative/liberal for some dependent hypotheses?

Question

I frequently read that Bonferroni correction also works for dependent hypotheses. However, I don't think that is true and I have a counter example. Can somebody please tell me (a) where my mistake is or (b) whether I am correct on this.

Setting up the counter example

Assume we are testing two hypotheses. Let $H_{1}=0$ is the first hypothesis is false and $H_{1}=1$ otherwise. Define $H_{2}$ similarly. Let $p_{1},p_{2}$ be the p-values associated with the two hypotheses and let $[\![\cdot]\!]$ denote the indicator function for the set specified inside the brackets.

For fixed $\theta\in [0,1]$ define \begin{eqnarray*} P\left(p_{1},p_{2}|H_{1}=0,H_{2}=0\right) & = & \frac{1}{2\theta}[\![0\le p_{1}\le\theta]\!]+\frac{1}{2\theta}[\![0\le p_{2}\le\theta]\!]\\ P\left(p_{1},p_{2}|H_{1}=0,H_{2}=1\right) & = & P\left(p_{1},p_{2}|H_{1}=1,H_{2}=0\right)\\ & = & \frac{1}{\left(1-\theta\right)^{2}}[\![\theta\le p_{1}\le1]\!]\cdot[\![\theta\le p_{2}\le1]\!] \end{eqnarray*} which are obviously probability densities over $[0,1]^{2}$. Here is a plot of the two densities

Marginalization yields \begin{eqnarray*} P\left(p_{1}|H_{1}=0,H_{2}=0\right) & = & \frac{1}{2\theta}[\![0\le p_{1}\le\theta]\!]+\frac{1}{2}\\ P\left(p_{1}|H_{1}=0,H_{2}=1\right) & = & \frac{1}{\left(1-\theta\right)}[\![\theta\le p_{1}\le1]\!] \end{eqnarray*} and similarly for $p_{2}$.

Furthermore, let \begin{eqnarray*} P\left(H_{2}=0|H_{1}=0\right) & = & P\left(H_{1}=0|H_{2}=0\right)=\frac{2\theta}{1+\theta}\\ P\left(H_{2}=1|H_{1}=0\right) & = & P\left(H_{1}=1|H_{2}=0\right)=\frac{1-\theta}{1+\theta}. \end{eqnarray*} This implies that \begin{eqnarray*} P\left(p_{1}|H_{1}=0\right) & = & \sum_{h_{2}\in\{0,1\}}P\left(p_{1}|H_{1}=0,h_{2}\right)P\left(h_{2}|H_{1}=0\right)\\ & = & \frac{1}{2\theta}[\![0\le p_{1}\le\theta]\!]\frac{2\theta}{1+\theta}+\frac{1}{2}\frac{2\theta}{1+\theta}+\frac{1}{\left(1-\theta\right)}[\![\theta\le p_{1}\le1]\!]\frac{1-\theta}{1+\theta}\\ & = & \frac{1}{1+\theta}[\![0\le p_{1}\le\theta]\!]+\frac{\theta}{1+\theta}+\frac{1}{1+\theta}[\![\theta\le p_{1}\le1]\!]\\ & = & U\left[0,1\right] \end{eqnarray*} is uniform as required for p-values under the Null hypothesis. The same holds true for $p_{2}$ because of symmetry.

To get the joint distribution $P\left(H_{1},H_{2}\right)$ we compute

\begin{eqnarray*} P\left(H_{2}=0|H_{1}=0\right)P\left(H_{1}=0\right) & = & P\left(H_{1}=0|H_{2}=0\right)P\left(H_{2}=0\right)\\ \Leftrightarrow\frac{2\theta}{1+\theta}P\left(H_{1}=0\right) & = & \frac{2\theta}{1+\theta}P\left(H_{2}=0\right)\\ \Leftrightarrow P\left(H_{1}=0\right) & = & P\left(H_{2}=0\right):=q \end{eqnarray*} Therefore, the joint distribution is given by \begin{eqnarray*} P\left(H_{1},H_{2}\right) & = & \begin{array}{ccc} & H_{2}=0 & H_{2}=1\\ H_{1}=0 & \frac{2\theta}{1+\theta}q & \frac{1-\theta}{1+\theta}q\\ H_{1}=1 & \frac{1-\theta}{1+\theta}q & \frac{1+\theta-2q}{1+\theta} \end{array} \end{eqnarray*} which means that $0\le q\le\frac{1+\theta}{2}$.

Why it is a counter example

Now let $\theta=\frac{\alpha}{2}$ for the significance level $\alpha$ of interest. The probability to get at least one false positive with the corrected significance level $\frac{\alpha}{2}$ given that both hypotheses are false (i.e. $H_{i}=0$) is given by \begin{eqnarray*} P\left(\left(p_{1}\le\frac{\alpha}{2}\right)\vee\left(p_{2}\le\frac{\alpha}{2}\right)|H_{1}=0,H_{2}=0\right) & = & 1 \end{eqnarray*} because all values of $p_{1}$ and $p_{2}$ are lower than $\frac{\alpha}{2}$ given that $H_1=0$ and $H_2=0$ by construction. The Bonferroni correction, however, would claim that the FWER is less than $\alpha$.

Answer 1

Bonferroni can't be liberal, regardless of dependence, if your p-values are computed correctly.

Let A be the event of Type I error in one test and let B be the event of Type I error in another test. The probability that A or B (or both) will occur is:

P(A or B) = P(A) + P(B) - P(A and B)

Because P(A and B) is a probability and thus can't be negative, there’s no possible way for that equation to produce a value higher than P(A) + P(B). The highest value the equation can produce is when P(A and B) = 0, i.e. when A and B are perfectly negatively dependent. In that case, you can fill in the equation as follows, assuming both nulls true and a Bonferroni-adjusted alpha level of .025:

P(A or B) = P(A) + P(B) - P(A and B) = .025 + .025 - 0 = .05

Under any other dependence structure, P(A and B) > 0, so the equation produces a value even smaller than .05. For example, under perfect positive dependence, P(A and B) = P(A), in which case you can fill in the equation as follows:

P(A or B) = P(A) + P(B) - P(A and B) = .025 + .025 - .025 = .025

Another example: under independence, P(A and B) = P(A)P(B). Hence:

P(A or B) = P(A) + P(B) - P(A and B) = .025 + .025 - .025*.025 = .0494

As you can see, if one event has a probability of .025 and another event also has a probability of .025, it’s impossible for the probability of “one or both” events to be greater than .05, because it’s impossible for P(A or B) to be greater than P(A) + P(B). Any claim to the contrary is logically nonsensical.

"But that's assuming both nulls are true," you might say. "What if the first null is true and the second is false?" In that case, B is impossible because you can't have a Type I error where the null hypothesis is false. Thus, P(B) = 0 and P(A and B) = 0. So let's fill in our general formula for the FWER of two tests:

P(A or B) = P(A) + P(B) - P(A and B) = .025 + 0 - 0 = .025

So once again the FWER is < .05. Note that dependence is irrelevant here because P(A and B) is always 0. Another possible scenario is that both nulls are false, but it should be obvious that the FWER would then be 0, and thus < .05.

Answer 2

I think I finally have the answer. I need an additional requirement on the distribution of $P(p_1,p_2|H_1=0, H_2=0)$. Before, I only required that $P(p_1|H_1=0)$ is uniform between 0 and 1. In this case my example is correct and Bonferroni would be too liberal. However, if I additionally require the uniformity of $P(p_1|H_1=0, H_2=0)$ then it is easy to derive that Bonferroni can never be too conservative. My example violates this assumption. In more general terms, the assumption is that the distribution of all p-values given that all null hypotheses are true must have the form of a copula: Jointly they don't need to be uniform, but marginally they do.

Comment: If anyone can point me to a source where this assumption is clearly stated (textbook, paper), I'll accept this answer.