Choosing the scope when performing multiple comparisons?

Question

On both a practical and philosophical level, how should you choose the scope when performing multiple comparisons?

When a study performs 10 tests to check the hypothesis that 10 explanatory variable are predictive for "something" (on the same dataset), the test should obviously be corrected.

What if there where ten studies, each testing for one different explanatory variable - when doing a meta analysis, should their P values be corrected? (will knowing if these studies where done on the same dataset, or on different datasets make a difference?)

But then, what if we add to the mix another 100 researchers, all of them where just not very good at their jobs (all where testing "junk" variables) - automatically that will ruin our chances at finding anything after correction. But is that a reflection of something already happening in real life science?

Now, let's assume the same researchers is doing a hundred studies, on different fields, asking one question in each of them. Should he have corrected his P values from these 100 studies? What if the questions are different but on the same study/dataset?

What are criterions would you offer for choosing the scoping of performing multiple comparisons correction?

p.s: I understand my question relates to this one, but since there are new people on the site, and since there is somewhat of a difference, I allowed myself to ask the above question.

p.p.s: I don't think this question has a "right answer", thus I choose to have it a community wiki, but for some reason I can't find how to do it in the screen today...

Answer 1

Think of the following two experiments:

Experiment A; Throw a fair coin 10 times to assess Prob(Heads).

Experiment B: Throw a fair dice 5 times to assess Prob(Face showing 1).

To take the coin toss example from the wiki: We may wish to declare a coin as biased if we observe more than 9 heads out of 10 tosses.

Thus, if I were to repeat experiment A 100 times then there is 34% chance (see the wiki for the calculations) that we would identify a coin as biased when it is not thus increasing out type I error probability from 0.05 to 0.34. Therefore, we need to control for multiple comparisons in this context.

However, note that our trials in experiment A have no influence on our results as far as experiment B is concerned as that is a completely different data generating process. The above suggests that we have to control for multiple comparisons for the two experiments separately instead of collectively.

In other words, controlling for multiple comparisons should be done whenever the comparisons involve the same data generating process.

Edit

Strictly speaking the above example of coin vs dice is not a good example as that would be analogous to experiments that investigate two very different questions (e.g., estimate whether smoking causes cancer and estimate if jumping red lights leads to an accident). In these contexts, controlling for multiple comparisons collectively for the two experiments is meaningless.

On further thinking, it is not clear to me if the data generating process really has a special role to play as far as multiple comparisons are concerned. Even if the data generating process were to be different (perhaps because of different covariates) you will still run the risk of increasing type I error because of multiple comparisons.

Therefore, it seems to me that what matters is whether the multiple comparisons involve making judgement about the same null hypothesis. As long as the multiple comparisons involve the same null hypothesis we have to correct for multiple comparisons to keep Type I error at desired levels (e.g., 0.05).