Pre-test probability and hypothesis testing

Question

I recently came across the article Statistical Errors, written by Regina Nuzzo (Nature, Feb 2014). I hope it is OK to include the image published in that article, as my question is directly linked to it:

I was wondering where those values come from. Say $H_1$ is the hypothesis that there is a real effect; $H_0$ means there is no effect. Let's say $P(\mathrm{eff})$ is the probability for an effect to exist. Further, $P(H_1)$ is the probability of the test to reject the null hypothesis and $P(H_0)$ the probability to accept the null hypothesis.

For the left-most example, I would now assign the following probabilities:

• $P(\mathrm{eff})=0.05$ and $P(\overline{\mathrm{eff}})=0.95$
• $P(H_1\mid\overline{\mathrm{eff}})\leq0.05$
• $P(\mathrm{eff}\mid H_1)=0.11$ and $P(\overline{\mathrm{eff}}\mid H_1=0.89$

Now with Bayes' theorem, I could conclude $$ P(\mathrm{eff}\mid H_1) = \frac{P(H_1\mid\overline{\mathrm{eff}})\cdot P(\overline{\mathrm{eff}})}{P(H_1)} $$ but $P(H_1)$ is unknown. I now thought I could use the law of total probability: $$ P(H_1) = P(H_1\mid\mathrm{eff})\cdot P(\mathrm{eff}) + P(H_1\mid\overline{\mathrm{eff}})\cdot P(\overline{\mathrm{eff}})$$ However, in this case, there is $P(H_1\mid\mathrm{eff})$ that I do not know. Thus, the snake is somehow biting its own tail.

How can I find the missing piece of information? Or what am I doing wrong?

Answer 1

It is not obvious what has been done in that chart, but the numbers might be consistent with $P(\text{report low }p\mid H_1)$ about $0.12$ when using the $p<0.05$ test and about $0.08$ when using the $p<0.01$ test (a test with fewer false positives can lead to fewer true positives too, so this makes some sort of sense), plus rounding to two decimal places.

In detail, the arithmetic for $$P(H_1\mid \text{report low }p) = \dfrac{P(\text{report low }p\mid H_1)P(H_1)}{P(\text{report low }p) } \\ =\dfrac{P(\text{report low }p\mid H_1)P(H_1)}{P(\text{report low }p\mid H_1)P(H_1)+P(\text{report low }p\mid H_0)P(H_0) }$$ seems to have been something like:

• $\dfrac{0.05 \times 0.12}{0.05 \times 0.12 + 0.95\times 0.05 } \approx 0.112$
• $\dfrac{0.05 \times 0.08}{0.05 \times 0.08 + 0.95\times 0.01 } \approx 0.296$
• $\dfrac{0.50 \times 0.12}{0.50 \times 0.12 + 0.50\times 0.05 } \approx 0.706$
• $\dfrac{0.50 \times 0.08}{0.50 \times 0.08 + 0.50\times 0.01 } \approx 0.889$
• $\dfrac{0.90 \times 0.12}{0.90 \times 0.12 + 0.10\times 0.05 } \approx 0.956$
• $\dfrac{0.90 \times 0.08}{0.90 \times 0.08 + 0.10\times 0.01 } \approx 0.986$