Imagine that you have a data set that is like the one below. (The data is all made up and is not specific to the problem.)

trial   n_control   n_intervention   n_control_responded   n_intervention_responded
1       497         1200             54                    91
2       307         908              22                    51
3       552         2042             150                   783
...     ...         ...              ...                   ...
100     2000        11303            713                   3308

Each row represents data about a trial that was conducted, in which an intervention was tried against a control group, and the response in each group was assessed. For the sake of the question, it can be assumed that each trial was done on a distinct sample (no overlap), that the trials were run at different times (not simultaneously), and tested distinct interventions (no duplication).

It is trivial to perform a statistical test on each of these rows and to return a p-value for each one. That p-value could be used to define a success or a failure of that intervention.

My question is: must the p-values which are returned be penalized in some fashion because of the multiple comparison problem? On the surface, the answer to this question is, of course, yes. If you do multiple tests, it is highly likely that one of them will come back as significant due to chance alone.

As is probably obvious, this format at least partially resembles the concept of a meta-analysis. It seems incorrect to say that we should operate on the assumption that, for instance, one out of every twenty papers published with a result of p < 0.05 is wrong. Similarly, it seems far-fetched to think that, as the literature expands, we should have increasingly stringent criteria for statistical significance, until ultimately, we have no ability at all to find anything. It fundamentally seems wrong to state that we should penalize the p-value of a set of unrelated studies in the name of controlling for multiple comparisons. But I have not found much guidance on the questions of how to determine when a set of tests are "different enough" to not require controlling for multiple comparisons.

If multiple comparisons corrects for random chance error, how applicable is it to cases where the multiple tests were performed in distinct experiments, on distinct samples, at distinct times? Put differently, what is the relationship between meta-analysis and multiple comparisons - are all meta-analyses automatically subject to the multiple comparison problem, or is there some standard which makes clear the difference between these two?


1 Answer 1


The focus of the methods is slightly different although they are of course not completely unconnected. The meta-analysis of significance values has as its mull hypothesis that all the $p_i$ are drawn from a uniform distribution on (0, 1). The alternative hypothesis is often not clearly stated but is usually that either (a) at least one of the $p_i$ has some other distribution, or (b) all of them have. People who adjust for multiple comparisons usually have a focus on which test has reached their corrected level of significance (or what the coverage of their adjusted confidence intervals is).

To put it another way meta-analysis here answers the question: taking the totality of the dataset together is there any evidence that all the $p_i$ are not uniform? The multiple comparisons approach answers the question: are these specific comparisons beyond the critical value allowing for the number of comparisons made.

  • $\begingroup$ How would you determine which method is more appropriate, though? It seems as though I could squint hard and justify either one. $\endgroup$
    – TARehman
    Commented Jun 20, 2016 at 17:18
  • $\begingroup$ @TARehman I have edited in some more explanation $\endgroup$
    – mdewey
    Commented Jun 21, 2016 at 11:51
  • $\begingroup$ So, in a sense, the question being answered is different. Meta-analysis asks whether the various experiments truly differ, but multiple comparison correction asks if the differences should be considered meaningful? $\endgroup$
    – TARehman
    Commented Jun 21, 2016 at 18:03
  • $\begingroup$ @TARehman that is not quite how I would have put it but I think I agree with you $\endgroup$
    – mdewey
    Commented Jun 22, 2016 at 8:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.