0
$\begingroup$

In Why Most Published Reasearch Findings Are False the author claims that the positive predictive value (PPV) of a $2\times2$ table can be expressed as:

$PPV= (1-\beta)\cdot R / ((1-\beta)\cdot R + \alpha) \,,$

where

  • $\alpha$: significance level
  • $1-\beta$: statistical power
  • $R$: pre-study odds for a probed effect to reflect a true effect

However, here is an example that prooves in my opinion that this formula is wrong: Suppose, I run $1\,000$ studies with $1-\beta = 0.3$ (a), $\alpha = 0.05$ (b) and $R= 0.2$ (c). Now, (c) gives us 200 wrong null hypotheses of which by (a) 60 are correctly identified. Among the remaining 800 correct null hypotheses 40 are wrongly rejected by (b). Thus, in total 100 null hypotheses are rejected of which 60 are correctly rejected, giving a PPV of 0.6. Had I instead used the above formula, the result would have been a PPV of

$PPV= 0.3 \cdot 0.2 / (0.3 \cdot 0.2 + 0.05) = 0.06 / 0.11 = 0.55$

So, in my opinion the correct formula should read:

$PPV= (1-\beta)\cdot R / ((1-\beta)\cdot R + \textbf{(1-R)} \cdot \alpha)$

Who is wrong?

EDIT:

Kbiolsi's remark is right: Ioannidis (at least in the above paper) defines $R$ as "the ratio of the number of 'true relationships' to 'no relationsships' amonge those tested in the field." And he further explains that "[t]he pre-study probability of a relationsip being true is R/(R+1)."

However, in Power failure: why small sample size undermines the reliability of neuroscience by Ioannidis et al. the above equation for PPV (the one I claimed to be wrong) is immediately followed by: "[...] R is the pre-study odds (that is, the odds that a probed effect is indeed non-null among the effects being probed)." Hence, Ioannidis clearly uses two differenct definitions of $R$ and (at least) in combination with this second definition the above equation is wrong.

Now, let me return to my above example. First, here is the requested contingency table:

enter image description here

What's the PVV in this situation? To my mind it is 0.6, which can be obtained by my above equation, assuming that $R$ is the pre-stuy odds for a probed effect to be true.

$\endgroup$
1
  • $\begingroup$ Can you write the table and explain the interpretation of the margins? $\endgroup$
    – AdamO
    Commented Sep 15, 2017 at 14:23

1 Answer 1

2
$\begingroup$

According to the original paper, R is the “ratio of the number of ‘true relationships’ to ‘no relationships’ among those tested in the field.” Since it is “ratio” and not “proportion”, if you have R=0.2 and have 200 true relationships, you will have 1000 (not 800) no relationships. So 50 correct null hypotheses are wrongly rejected for a total of 110 rejections, 60 of which are correct.

EDIT: I'll add to my answer to address your edit.

For the table you show, yes, PPV = 0.6. But, for this table, R=0.25 (200/800, the odds), not 0.2 (200/1000, the proportion of the total). Therefore, the computation using the Ioannidis formula is 0.3*0.25/(0.3*0.25+0.05) = 0.6, that is, the same value you get.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.