Actual number of false positive at α =0.05

Question

Forstmeier et al 2017 claim that when testing 1000 hypotheses with a p-value threshold of α =0.05 and a scenario where 10% of the hypotheses are true, 45 null hypotheses will be wrongfully rejected.

Consider a thousand hypotheses H1 that we might wish to test. Many of these may not be true, so let us start with a scenario where only 10% of the hypotheses at hand are in fact true. This proportion of hypotheses being true is often described with the symbol π (here π =0.1). When testing the 900 hypotheses that are not true, we allow for 5% false-positive findings if we set our significance threshold at α =0.05 (the accepted level of >making Type I errors). This means we will obtain 45 (i.e. >900×0.05) false-positive answers

1) Is this number of false-positives changed when considering that most p-values reported in science articles are below 0.05?

2) For example, if the p-values inferior to α =0.05 that are reported are about 0.001 on average, would we expect 1 false-positive result only?

Reference:

Forstmeier, Wolfgang, Eric‐jan Wagenmakers, and Timothy H. Parker. "Detecting and avoiding likely false‐positive findings–a practical guide." Biological Reviews 92.4 (2017): 1941-1968.

Answer 1

1) Yes and no. If the original scenario had used a lower rate then the number of false positives would fall as well. This article is just showing Bayes theorem.

2) Yes, approximately.

However, you may want to read

Ruud Wetzels, Dora Matzke, Michael D. Lee, Jeffrey N. Rouder, Geoffrey J. Iverson, and Eric-Jan Wagenmakers. Statistical Evidence in Experimental Psychology: An Empirical Comparison Using 855 t Tests. Perspectives on Psychological Science. 6(3) 291–298

They deconstruct 855 published results using Bayesian methods and while the results are generally the same, the Bayesian tests were less likely to find significance and in some cases found support for the null hypothesis, which is something a non-Bayesian test cannot do. Some findings that were statistically significant under a Frequentist t-test favored the null hypothesis when tested using Bayesian methods.

And, of course, there is the famous

Ioannidis JPA. Why Most Published Research Findings Are False. PLoS Medicine. 2005;2(8):e124. doi:10.1371/journal.pmed.0020124.

The difficulty is that most journals will not publish non-significant findings. This, in turn, means that a very large percentage of printed findings were due to chance.