On both a practical and philosophical level, how should you choose the scope when performing multiple comparisons?

When a study performs 10 tests to check the hypothesis that 10 explanatory variable are predictive for "something" (on the same dataset), the test should obviously be corrected.

What if there where ten studies, each testing for one different explanatory variable - when doing a meta analysis, should their P values be corrected? (will knowing if these studies where done on the same dataset, or on different datasets make a difference?)

But then, what if we add to the mix another 100 researchers, all of them where just not very good at their jobs (all where testing "junk" variables) - automatically that will ruin our chances at finding anything after correction. But is that a reflection of something already happening in real life science?

Now, let's assume the same researchers is doing a hundred studies, on different fields, asking one question in each of them. Should he have corrected his P values from these 100 studies? What if the questions are different but on the same study/dataset?

What are criterions would you offer for choosing the scoping of performing multiple comparisons correction?

p.s: I understand my question relates to this one, but since there are new people on the site, and since there is somewhat of a difference, I allowed myself to ask the above question.

p.p.s: I don't think this question has a "right answer", thus I choose to have it a community wiki, but for some reason I can't find how to do it in the screen today...

  • 2
    $\begingroup$ What would be the purpose of (or what would you conclude from) a meta-analysis on studies involving the same outcome but different predictors or risk factors? $\endgroup$
    – chl
    Nov 4 '10 at 22:19
  • 1
    $\begingroup$ Hi chl, If I where to send 100 researchers to study different risk factors for the same disease, and 5 of them came back with a P value bellow 5% (but not much less). Would it be relevant to somehow take all the studies into account? (I believe on the practical level, the answer is no. But on a more hypothetical level - I am not sure). From the little I know of meta analysis, it would be relevant only if there was an overlap in risk factors, otherwise, it doesn't offer much (at least from the little I know). Is that what you where aiming at? $\endgroup$
    – Tal Galili
    Nov 4 '10 at 23:34
  • 1
    $\begingroup$ Well, I was rather thinking of meta-analysis as a tool to pool measures of association (or ESs) across several studies, in a retrospective way, which means working with at least a common risk factor (most of the time adjusted for potential confounders). So your use of "meta-analysis" encompasses a much broader way of combining results from different independent studies (IMO). $\endgroup$
    – chl
    Nov 5 '10 at 8:36

Think of the following two experiments:

Experiment A; Throw a fair coin 10 times to assess Prob(Heads).

Experiment B: Throw a fair dice 5 times to assess Prob(Face showing 1).

To take the coin toss example from the wiki: We may wish to declare a coin as biased if we observe more than 9 heads out of 10 tosses.

Thus, if I were to repeat experiment A 100 times then there is 34% chance (see the wiki for the calculations) that we would identify a coin as biased when it is not thus increasing out type I error probability from 0.05 to 0.34. Therefore, we need to control for multiple comparisons in this context.

However, note that our trials in experiment A have no influence on our results as far as experiment B is concerned as that is a completely different data generating process. The above suggests that we have to control for multiple comparisons for the two experiments separately instead of collectively.

In other words, controlling for multiple comparisons should be done whenever the comparisons involve the same data generating process.


Strictly speaking the above example of coin vs dice is not a good example as that would be analogous to experiments that investigate two very different questions (e.g., estimate whether smoking causes cancer and estimate if jumping red lights leads to an accident). In these contexts, controlling for multiple comparisons collectively for the two experiments is meaningless.

On further thinking, it is not clear to me if the data generating process really has a special role to play as far as multiple comparisons are concerned. Even if the data generating process were to be different (perhaps because of different covariates) you will still run the risk of increasing type I error because of multiple comparisons.

Therefore, it seems to me that what matters is whether the multiple comparisons involve making judgement about the same null hypothesis. As long as the multiple comparisons involve the same null hypothesis we have to correct for multiple comparisons to keep Type I error at desired levels (e.g., 0.05).

  • 1
    $\begingroup$ Hello Srikant, thank you for thinking about this. Here is a counter example: Let's say we have done a research (which will be published) on which of a 100 risk factors effects a disease. In our article we have received 100 P values, 5 of which where around 5% ("significant"). The Null hypothesis would be different ("risk factor i is not effecting the disease"). But still if we won't correct for multiplicity, we will get E(x)=5 wrong "discoveries". So unless I'm missing something, relying on the null hypothesis, IMHO, won't be enough. $\endgroup$
    – Tal Galili
    Nov 5 '10 at 4:25
  • 1
    $\begingroup$ @Tal But, your null hypothesis is "None of the risk factors impact disease". This is evident when you talk about wrong discoveries. Thus, you are using the same null hypothesis when you are performing these multiple comparisons and hence the need for control. If on the other hand your null was: "Age has no impact on disease" then you would not be interested in a study that did not have age as a covariate and hence your multiple comparisons would be restricted to the studies where age was used as a covariate. Therefore, it seems to me that my original point is still valid in your example. $\endgroup$
    – user28
    Nov 5 '10 at 14:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.