Let's take an example. There is a huge hype around bananas and every medical researcher believes that bananas can cure cancer. 20 laboratories conduct a study to test whether or not bananas indeed cure cancer, they use a significance level of 5%. In reality, bananas do not cure cancer, it's strawberries. Therefore, on average, 19 out of those 20 laboratories will have insignificant results, and one laboratory will have a significant result. Individually, they all did their job properly, honestly. Nevertheless, on an aggregate level, it's like if they manipulated the results using multiple comparisons.

The question is: How can we control for this problem ("multiple comparisons across several researchers")?

I had two ideas to solve the problem:

  • Publish insignificant results: It could be a database where every study (even with insignificant results) gets reported. If you see that similar studies to yours have all failed in the past, you can suspect a false positive. Nevertheless, it doesn't seem to me that it can solve the problem of 19 other laboratories testing 19 different fruits that all have no effect, and you are the lucky 20th laboratory.
  • Re-run the test: Basically the rule for a significant result would be: run the test once, if it's significant then re-run it. I'm not particularly convinced of this one because it's costly, but also it will change the maths (you need to pass two tests in a row), and it might just move the problem one test further and not solve anything.

Do you have any other solution? Can you give arguments why my proposed solutions are either good or bad?

  • 1
    $\begingroup$ A good question requires some search and research - which would quickly reveal both of your proposals (publishing insignificant results and replication of studies) are often suggested. "Do you have any other solution" seems a bit broad / likely to invite a wide variety of speculative answers -- it would be better if you could consider a more focused question. $\endgroup$ – Glen_b Jun 11 '17 at 10:30

Your proposals are sound.

The first one is essentially trial pre-registration, which aims at reducing the file-drawer problem and publication bias.

Your second proposal is replication, which should be done more often. The problem is that replications are not as "interesting" as novel findings, so replications have a harder time getting funding and getting published in good journals.

A third possibility is meta-analysis, where you take multiple published studies on similar questions and summarize them, both statistically and scientifically. There are established tools to (attempt to) account for publication and similar biases in meta-analyses, such as funnel plots.


Another thing to consider is that research findings don't exist in a vacuum. Usually the hypothesis you test will generate more than one prediction, or interact with other theories on related or wider topics. So we can usually design follow-up experiments that test the same hypothesis from a different angle, which can lead to convergent evidence that is more convincing than a simple replication of the original design. In your example, an alternative to a pure replication could be an experiment that tries to isolate the active ingredient in bananas that cures cancer. As long as that is not much more expensive than a 1:1 replication, this may be a more efficient follow-up, because if the original result was true, the new experiment may not only reaffirm that result (by finding the ingredient) but simultaneously add to our knowledge (because we can now develop a cure based on the active substance we've discovered). Or, if this is too expensive, we might retry the original experiment with a fruit that is a close relative to the banana, or with a different part of the banana plant.

Also important is how we actually think about p-values and statistical significance. The criterion p < 0.05 was never intended to mean "we should assume this is true". The paper that originally proposed it simply meant it to be a threshold beyond which some effect should be considered significant enough for further study. As long as we treat it this way, we can protect ourselves against bad inferences. The problem is that currently, many people consider p < 0.05 to be a sort of "finish line" for science, while really it should just be the beginning.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.