Let's take an example. There is a huge hype around bananas and every medical researcher believes that bananas can cure cancer. 20 laboratories conduct a study to test whether or not bananas indeed cure cancer, they use a significance level of 5%. In reality, bananas do not cure cancer, it's strawberries. Therefore, on average, 19 out of those 20 laboratories will have insignificant results, and one laboratory will have a significant result. Individually, they all did their job properly, honestly. Nevertheless, on an aggregate level, it's like if they manipulated the results using multiple comparisons.
The question is: How can we control for this problem ("multiple comparisons across several researchers")?
I had two ideas to solve the problem:
- Publish insignificant results: It could be a database where every study (even with insignificant results) gets reported. If you see that similar studies to yours have all failed in the past, you can suspect a false positive. Nevertheless, it doesn't seem to me that it can solve the problem of 19 other laboratories testing 19 different fruits that all have no effect, and you are the lucky 20th laboratory.
- Re-run the test: Basically the rule for a significant result would be: run the test once, if it's significant then re-run it. I'm not particularly convinced of this one because it's costly, but also it will change the maths (you need to pass two tests in a row), and it might just move the problem one test further and not solve anything.
Do you have any other solution? Can you give arguments why my proposed solutions are either good or bad?