4
$\begingroup$

Suppose I have a set of tests, represented by their (uncorrected) p-values: $T = \{p_1, \dots,p_m \}$. These could be, for example, p-values from $m$ ANOVAs, performed on linear regressions with $d$ coefficients each.

The underlying linear regressions also have, each, a set of $d$ p-values attached to them (if I want to test for the coefficients, for example): $S_i=\{p'_1,...,p'_d\}$. These are the effects of interest.

To keep the number of test for a minimum (and optimize power), I would like to only assess significance of the sets $S_i$ that had significant $p_i$.

How can I correctly perform the correction for multiple comparisons in this scenario, for either FWER or FDR? It's reminiscent of step-down procedures I guess, where correction of further p's depends on the significance of earlier ones. I also see relations to post-hoc pairwise-testing procedures, but I couldn't find anything related to this.

  1. Should I correct for multiple comparisons the initial, omnibus tests, $T$, and then correct only the corresponding $S_i$'s? (this increases power, but I'm not sure it keeps FWER/FDR under control, because I'm not taking into account the $T$'s when correcting $S$'s)
  2. Should I jointly correct $S$ and $T$ before assessing significance? (this surely decreases power though, because the number of tests increases this way)
  3. Should I not correct $T$, and then correct subtests in the $S_i$'s corresponding to the significant $T$'s? (This is similar to #1)
  4. Any other alternatives?
$\endgroup$

1 Answer 1

1
$\begingroup$

It came to my attention this type of procedure has been described elsewhere. You'll find it under two-step, hierarchical, hypothesis testing.

I'll reproduce the method described in Li and Ghosh, 2014.

Given $m$ sets of hypotheses $\left\{H(1),\ldots,H(m)\right\}$ which we want to test simultaneously.

In each set, there's a screening hypothesis, denoted by the $H_0(i)$ null-hypothesis. The other hypothesis in the set will only be tested if the first one is rejected $H_0(i)$.

The procedure consists of:

  1. Correct for multiple comparisons (Benjamini-Hochberg FDR, in the study) the p-values associated with all screening tests and store the number of rejections $R$.
  2. For each set where a rejection occurred, control for the family-wise error rate at the $R\alpha/m$ level.

Authors posit that this simple procedure controls the Overall FDR (OFDR) across sets of hypothesis. However, the procedure requires that individual non-screening hypothesis in each set are independent of all screening hypothesis in other sets.


Li, Yihan, and Debashis Ghosh. “A Two-Step Hierarchical Hypothesis Set Testing Framework, with Applications to Gene Expression Data on Ordered Categories.” BMC Bioinformatics 15, no. 1 (2014). https://doi.org/10.1186/1471-2105-15-108.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.