Consider the following gatekeeping strategy (for the sake of simplicity, let's assume that family $F_i$ has only 1 hypothesis, $H_i$),

  • Hypothesis $H_1$ is tested at the 5% significance level,
  • If $H_1$ cannot be reject, then stop
  • Otherwise, $H_2$ is tested at the 5% significance level,
  • And so forth.

Why/How does this procedure maintain the type 1 error rate at 5%? I.e.,

$\Pr(\text{reject } H_1 \text{ or reject } H_2 \,|\, H_1 \text{ is true and } H_2 \text{ is true}) = 0.05$

enter image description here

Source: https://statmodeling.stat.columbia.edu/wp-content/uploads/2017/11/jama_Yadav_2017_gm_170003.pdf

  • $\begingroup$ It doesn't (look like). Can you give us some context? What are these hypotheses? What data are they being tested on? $\endgroup$ Commented Sep 5, 2023 at 8:50
  • $\begingroup$ That's my understanding of statmodeling.stat.columbia.edu/wp-content/uploads/2017/11/…. Post edited accordingly $\endgroup$
    – user7064
    Commented Sep 5, 2023 at 8:53
  • $\begingroup$ I don't buy this argument. One of the key steps of this procedure is to order the hypotheses accordingly (from best to worst in a sense), which is just a different way of doing multiple comparisons before doing multiple comparisons. $\endgroup$ Commented Sep 5, 2023 at 9:01
  • $\begingroup$ @user2974951: Could you please make it an answer with the more detailed procedure and the calculation of the type I error? $\endgroup$
    – user7064
    Commented Sep 5, 2023 at 10:06

1 Answer 1


This is easy to show by re-writing this as a closed testing procedure, which says

  • the intersection of the two null hypotheses is rejected iff $p_1 \leq \alpha$,
  • the first elementary null hypothesis is rejected iff $p_1 \leq \alpha$, and
  • the second elementary null hypothesis is rejected iff $p_2 \leq \alpha$.

Of course, this gets increasingly annoying to do the more elementary null hypotheses are involved (you have to consider the intersections of all possible combinations of elementary null hypotheses). That's why people came up with a simpler ways of doing this. In practice, the way I would usually rely on exploits the fact that if you can display this procedure as a graphical sequentially rejective multiple test procedures:

Graphical testing procedure version

If such a procedure has total initially assigned $\alpha$ smaller than the intended level of the test and all outgoing vertices having weights than sum to less than 1, then you know that this translates into a closed testing procedure that controls the familywise type I error rate at level $\alpha$ (see the linked paper).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.