Imagine that you have a data set that is like the one below. (The data is all made up and is not specific to the problem.)
trial n_control n_intervention n_control_responded n_intervention_responded 1 497 1200 54 91 2 307 908 22 51 3 552 2042 150 783 ... ... ... ... ... 100 2000 11303 713 3308
Each row represents data about a trial that was conducted, in which an intervention was tried against a control group, and the response in each group was assessed. For the sake of the question, it can be assumed that each trial was done on a distinct sample (no overlap), that the trials were run at different times (not simultaneously), and tested distinct interventions (no duplication).
It is trivial to perform a statistical test on each of these rows and to return a p-value for each one. That p-value could be used to define a success or a failure of that intervention.
My question is: must the p-values which are returned be penalized in some fashion because of the multiple comparison problem? On the surface, the answer to this question is, of course, yes. If you do multiple tests, it is highly likely that one of them will come back as significant due to chance alone.
As is probably obvious, this format at least partially resembles the concept of a meta-analysis. It seems incorrect to say that we should operate on the assumption that, for instance, one out of every twenty papers published with a result of
p < 0.05 is wrong. Similarly, it seems far-fetched to think that, as the literature expands, we should have increasingly stringent criteria for statistical significance, until ultimately, we have no ability at all to find anything. It fundamentally seems wrong to state that we should penalize the p-value of a set of unrelated studies in the name of controlling for multiple comparisons. But I have not found much guidance on the questions of how to determine when a set of tests are "different enough" to not require controlling for multiple comparisons.
If multiple comparisons corrects for random chance error, how applicable is it to cases where the multiple tests were performed in distinct experiments, on distinct samples, at distinct times? Put differently, what is the relationship between meta-analysis and multiple comparisons - are all meta-analyses automatically subject to the multiple comparison problem, or is there some standard which makes clear the difference between these two?