On both a practical and philosophical level, how should you choose the scope when performing multiple comparisons?
When a study performs 10 tests to check the hypothesis that 10 explanatory variable are predictive for "something" (on the same dataset), the test should obviously be corrected.
What if there where ten studies, each testing for one different explanatory variable - when doing a meta analysis, should their P values be corrected? (will knowing if these studies where done on the same dataset, or on different datasets make a difference?)
But then, what if we add to the mix another 100 researchers, all of them where just not very good at their jobs (all where testing "junk" variables) - automatically that will ruin our chances at finding anything after correction. But is that a reflection of something already happening in real life science?
Now, let's assume the same researchers is doing a hundred studies, on different fields, asking one question in each of them. Should he have corrected his P values from these 100 studies? What if the questions are different but on the same study/dataset?
What are criterions would you offer for choosing the scoping of performing multiple comparisons correction?
p.s: I understand my question relates to this one, but since there are new people on the site, and since there is somewhat of a difference, I allowed myself to ask the above question.
p.p.s: I don't think this question has a "right answer", thus I choose to have it a community wiki, but for some reason I can't find how to do it in the screen today...