1
$\begingroup$

I recently came across the article Statistical Errors, written by Regina Nuzzo (Nature, Feb 2014). I hope it is OK to include the image published in that article, as my question is directly linked to it:enter image description here

I was wondering where those values come from. Say $H_1$ is the hypothesis that there is a real effect; $H_0$ means there is no effect. Let's say $P(\mathrm{eff})$ is the probability for an effect to exist. Further, $P(H_1)$ is the probability of the test to reject the null hypothesis and $P(H_0)$ the probability to accept the null hypothesis.

For the left-most example, I would now assign the following probabilities:

  • $P(\mathrm{eff})=0.05$ and $P(\overline{\mathrm{eff}})=0.95$
  • $P(H_1\mid\overline{\mathrm{eff}})\leq0.05$
  • $P(\mathrm{eff}\mid H_1)=0.11$ and $P(\overline{\mathrm{eff}}\mid H_1=0.89$

Now with Bayes' theorem, I could conclude $$ P(\mathrm{eff}\mid H_1) = \frac{P(H_1\mid\overline{\mathrm{eff}})\cdot P(\overline{\mathrm{eff}})}{P(H_1)} $$ but $P(H_1)$ is unknown. I now thought I could use the law of total probability: $$ P(H_1) = P(H_1\mid\mathrm{eff})\cdot P(\mathrm{eff}) + P(H_1\mid\overline{\mathrm{eff}})\cdot P(\overline{\mathrm{eff}})$$ However, in this case, there is $P(H_1\mid\mathrm{eff})$ that I do not know. Thus, the snake is somehow biting its own tail.

How can I find the missing piece of information? Or what am I doing wrong?

$\endgroup$

1 Answer 1

0
$\begingroup$

It is not obvious what has been done in that chart, but the numbers might be consistent with $P(\text{report low }p\mid H_1)$ about $0.12$ when using the $p<0.05$ test and about $0.08$ when using the $p<0.01$ test (a test with fewer false positives can lead to fewer true positives too, so this makes some sort of sense), plus rounding to two decimal places.

In detail, the arithmetic for $$P(H_1\mid \text{report low }p) = \dfrac{P(\text{report low }p\mid H_1)P(H_1)}{P(\text{report low }p) } \\ =\dfrac{P(\text{report low }p\mid H_1)P(H_1)}{P(\text{report low }p\mid H_1)P(H_1)+P(\text{report low }p\mid H_0)P(H_0) }$$ seems to have been something like:

  • $\dfrac{0.05 \times 0.12}{0.05 \times 0.12 + 0.95\times 0.05 } \approx 0.112$
  • $\dfrac{0.05 \times 0.08}{0.05 \times 0.08 + 0.95\times 0.01 } \approx 0.296$
  • $\dfrac{0.50 \times 0.12}{0.50 \times 0.12 + 0.50\times 0.05 } \approx 0.706$
  • $\dfrac{0.50 \times 0.08}{0.50 \times 0.08 + 0.50\times 0.01 } \approx 0.889$
  • $\dfrac{0.90 \times 0.12}{0.90 \times 0.12 + 0.10\times 0.05 } \approx 0.956$
  • $\dfrac{0.90 \times 0.08}{0.90 \times 0.08 + 0.10\times 0.01 } \approx 0.986$
$\endgroup$
3
  • $\begingroup$ Thanks for that answer. Just to be sure: The information I claimed was missing, was really missing and should have been stated somewhere and you found those 0.12 and 0.08 by reverse engineering the illustration, right? $\endgroup$ Oct 27, 2020 at 16:43
  • $\begingroup$ @AnonymousStatistician Yes - I think so. The article itself is at nature.com/news/scientific-method-statistical-errors-1.14700 and the version of the chart there seems to mention tandfonline.com/doi/abs/10.1198/000313001300339950 but I cannot see this $0.12$ and $0.08$ stated $\endgroup$
    – Henry
    Oct 27, 2020 at 17:08
  • $\begingroup$ Thanks again. For as much as I like the idea behind the article and the chart, I consider this a weak point. Even more so, because IMHO $P(\mathrm{report low $p$})=0.12$ and thus a type II error probability of 0.88 seems very arbitrary. The results would be completely with a test that has a higher sensitivity. Basically, she is just choosing values that support her claim, but does not even make it transparent. $\endgroup$ Oct 27, 2020 at 20:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.