# Why do we not always use the closed testing principle for multiple comparisons?

From the Wikipedia entry:

Suppose there are k hypotheses $$H_1,..., H_k$$ to be tested and the overall type I error rate is $$\alpha$$. The closed testing principle allows the rejection of any one of these elementary hypotheses, say $$H_i$$, if all possible intersection hypotheses involving $$H_i$$ can be rejected by using valid local level $$\alpha$$ tests. It controls the familywise error rate for all the k hypotheses at level α in the strong sense.

By using the closed testing principle, we ensure all the false positive rate of all hypothesis tests in a family will be controlled at $$\alpha$$. This seems too good to be true. Why do we bother with Bonferonni or other corrections for multiple comparisons, when we can just test all the intersection hypotheses? The only downside seems to be that you need to conduct $$2^{n}$$ tests.

EDIT: I am thinking about a relatively small number of individual hypothesis tests, say $$n = 3 \text{or} 5$$. Then you would have a reasonable number of tests.

• Your "only" downside is a huge one! It limits practical procedures to $n\ll 30$ or so, and in cases where the tests are onerous to compute (think bootstrapping, for instance) to $n\ll 10.$ – whuber Sep 27 '19 at 13:33
• @whuber Not really true, is it? The closed testing principle is used all the time. – Björn Sep 27 '19 at 13:39
• @whuber, I had in mind a relatively small number of hypothesis tests. For example, I'm curious why didn't I learn this for ANOVA with two treatments and an interaction. – EliK Sep 27 '19 at 13:41
• I wonder if this sucks power from the test. Sure, you control $\alpha$, but at what cost? If even the Bonferroni correction is more powerful, then I think that this would be an inadmissable testing procedure. (Confession: I don't know this to be the case, and it might be totaly wrong.) – Dave Sep 27 '19 at 13:53
• @Dave I don't think Bonferroni is more powerful, but I'm not sure. – EliK Sep 27 '19 at 14:12

It is actually used quite often (in fact, Bonferroni and Bonferroni-Holm can be shown to be valid tests using the closed testing principle - see below). Part of the reason why some simple procedures like Bonferroni are still so popular is of course, that they are very easy to implement and it is easy to communicate what you did.

E.g. even for the simple Bonferroni or Bonferroni-Holm writing down the tests for the intersection null hypotheses is a bit tedious. That's why the graphical approach to constructing closed testing procedures is very popular - at least in the clinical trials setting (where there is usually a trained statistician at hand that can help pre-specify a tailored testing procedure for a specific trial). This graphical approach provides a really easy way to construct valid testing procedures and makes communication easier. Various variants, extensions and improvements such as exploiting correlations, prioritization of certain hypotheses depending on which hypotheses are already rejected etc. have been proposed over the years.

Bonferroni vs. Bonferroni-Holm as closed testing procedures example

The Bonferroni test can be easily written down in terms of the closed testing principle, e.g. in the case of two null hypotheses as

• Test for $$H_1 \cap H_2$$: $$p_1 \leq \alpha/2$$ and $$p_2 \leq \alpha/2$$
• Test for elementary null hypothesis $$H_1$$: $$p_1 \leq \alpha/2$$
• Test for elementary null hypothesis $$H_2$$: $$p_2 \leq \alpha/2$$

with $$H_1$$ rejected overall, if both $$H_1$$ and $$H_1 \cap H_2$$ are rejected. Thus, we know it is a valid test. You also immediately see that for the elementary null hypotheses $$H_1$$ and $$H_2$$ you are using a test that is not exhausting the significance level and that the Bonferroni-Holm test

• Test for $$H_1$$ and $$H_2$$: $$p_1 \leq \alpha/2$$ and $$p_2 \leq \alpha/2$$
• Test for $$H_1$$: $$p_1 \leq \alpha$$
• Test for $$H_2$$: $$p_2 \leq \alpha$$

is uniformly more powerful.

Figure 1: Illustration of the graphical approach for displaying these two procedures Power considerations and other considerations

Usually, there's no downside in terms of power to using a closed testing procedure - if anything it tends to be easier to make sure the test fully exhausts the significance level. It can sometimes be a bit tricky to build in some desired features (e.g. exploiting a known correlation) while keep some other desired contraints (e.g. that the closed testing procedure can be easily displayed graphically due to the consistency requirement on the testing procedure).

• Thank you. I have a followup question. Let's ignore changing the importance of particular test or accounting for correlation. If I can construct a uniformally most powerful test (UMP) for every hypothesis test in the family, is that the most powerful procedure I can construct that still controls the Family wise Error Rate. – EliK Sep 27 '19 at 14:18