2
$\begingroup$

This might be a rather basic question, but I really want to get it right.

When doing multiple hypothesis testing, researchers typically account for the "multiple" part by controlling for the family wise error rate by means of e.g. the Bonferroni Correction procedure.

However, is it possible to make a "summarizing" inference based on the hypothesis test results of multiple tests? Say, for example, I conducted a hypothesis test repeatedly for 5 times. I controlled for the family wise error rate by adjusting each p-value by applying the Bonferroni Correction procedure. In 3 cases I reject the null hypothesis, in 2 cases I do not.

Can I reject or accept the null hypothesis "in general" based on this result? I.e. can I summarize those 5 test results in some way?

$\endgroup$
2
  • 1
    $\begingroup$ An omnibus null hypothesis, for example, of the form $\text{H}_{0}\text{: }\theta_{i} = \theta_{j}$, for all $i \ne j$, and $i, j$ index groups $1$ through $k$, where $\text{H}_{\text{A}}\text{: }\theta_{i} \ne \theta_{j}$ for at least one $i \ne j$ summarizes differences across all pairwise combinations of $k$ groups. For example, one-way ANOVA, Kruskal-Wallis, etc. are omnibus tests. Are you asking about this? $\endgroup$ – Alexis Jun 15 at 2:04
  • 1
    $\begingroup$ Not exactly what I was looking for, but it still helps - thanks! $\endgroup$ – shenflow Jun 15 at 10:09
2
$\begingroup$

Often, a "summary" test refers to a test of a global hypothesis. In your case, if your global hypothesis is "All five nulls are true," and then if you reject at least one hypothesis using the Bonferroni criterion, then you have successfully rejected the global hypothesis. Often a (summary) p-value is attached to such a test; in your case the p-value would be the Bonferroni-adjusted p-value for the most significant result; i.e., $p_{adjusted} = 5\min p_i$, where $i = 1,2,3,4,5$, and the $p_i$ refer to the ordinary (unadjusted) p-values for each of the five tests.

This method generalizes easily to account for dependence structures and non-normalities; see e.g. https://www.amazon.com/Resampling-Based-Multiple-Testing-Examples-Adjustment/dp/0471557617

$\endgroup$
2
  • $\begingroup$ Another global hypothesis is that at least one of the nulls are true. $\endgroup$ – Galen Jun 15 at 2:00
  • $\begingroup$ Thanks to both of you. $\endgroup$ – shenflow Jun 15 at 10:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.