Forstmeier et al 2017 claim that when testing 1000 hypotheses with a p-value threshold of α =0.05 and a scenario where 10% of the hypotheses are true, 45 null hypotheses will be wrongfully rejected.
Consider a thousand hypotheses H1 that we might wish to test. Many of these may not be true, so let us start with a scenario where only 10% of the hypotheses at hand are in fact true. This proportion of hypotheses being true is often described with the symbol π (here π =0.1). When testing the 900 hypotheses that are not true, we allow for 5% false-positive findings if we set our significance threshold at α =0.05 (the accepted level of >making Type I errors). This means we will obtain 45 (i.e. >900×0.05) false-positive answers
1) Is this number of false-positives changed when considering that most p-values reported in science articles are below 0.05?
2) For example, if the p-values inferior to α =0.05 that are reported are about 0.001 on average, would we expect 1 false-positive result only?
Forstmeier, Wolfgang, Eric‐jan Wagenmakers, and Timothy H. Parker. "Detecting and avoiding likely false‐positive findings–a practical guide." Biological Reviews 92.4 (2017): 1941-1968.