In our world, only statistically significant ($p<0.05$) results are published, everything else is rejected or not even handed in.

Let's now assume an experiment is $100$ times conducted and in truth $H_0$ holds. Since the $p$-value is uniformly distributed under $H_0$ (let's assume all conditions are satisfied) this would mean that on average $5$-paper could be handed in which all claim that they have found a statistically significant effect ($p<0.05$) and the academic world might actually believe in the alternative hypothesis $H_1$ although $H_0$ is actually true (of course no one would actually know that).

Question: Is this reasoning correct? I wonder how this reporting bias is called, it's different from the publication bias (or just a consequence and that's why it has name of its own?).

  • $\begingroup$ These issues are discussed extensively on this page. $\endgroup$
    – EdM
    Jan 13 '20 at 16:42
  • $\begingroup$ @EdM thanks! I'd say it answers my question more or less - definitely helpful to anyone who is also having this kind of question! $\endgroup$
    – user190080
    Jan 13 '20 at 18:03

This scenario seems like a straightforward example of publication bias. The definition at the linked Wikpedia article is a good one: publication bias "occurs when the outcome of an experiment or research study influences the decision whether to publish or otherwise distribute it." In your scenario, the 95 experiments that failed to reject the null hypothesis are not published whereas the 5 that succeeded are.


This effect is a form of publication bias, but also an effect of wholesale adoption of significance at P $\leq$ 0.05, ignoring effect size, and the crisis of reproducibility in science.

I add the alpha level of 0.05, as this level is not particularly stringent for the reason the OP states - 5/100 studies will be statistically significant by chance alone.

Effect size is often ignored which elevates trivial effects when p-values are small and may ignore important effects when p-values are large. Read here for more on reproducibility:

Baker, Monya. "Is there a reproducibility crisis? A Nature survey lifts the lid on how researchers view the 'crisis rocking science and what they think will help." Nature, vol. 533, no. 7604, 2016, p. 452+. Gale OneFile: Health and Medicine, Accessed 13 Jan. 2020.

In aggregate, science doesn’t publish all work equally, abuses p-values at the cost of effect size and doesn’t/can not verify the direction or exent of biases created.


Not the answer you're looking for? Browse other questions tagged or ask your own question.