Statistical significance and the two-trials rule in clinical trials

Question

If the false discovery rate is known to be 0.40, how do we obtain the probability that two consecutive statistically significant results arising from an application of the two-trials rule are true positives?

Answer 1

You can't, without more information, because the distribution of true effect sizes is more complicated than 'zero' or 'not zero' and affects this probability.

Even under the toy model that all trials either have a zero effect size or an effect size of exactly $\delta$ and all trials have the same power $q$ at an effect size of $\delta$, and the same probability $p$ of a non-zero effect, and the tests have their correct size, you could get a false discovery rate of 40% with small $p$ and larger $q$ or large $p$ and small $q$. With high power, the second study is much more likely to be positive when the first study was a true discovery; with low power, not so much.

Let $\alpha=0.025$ be the size of the test. We have $pq+(1-p)\alpha$ positive results, with a false discovery rate of $$\frac{(1-p)\alpha}{(pq+(1-p)\alpha)} =0.4$$ so $$p=\frac{0.6\alpha}{0.6\alpha+0.4q}$$ The probability of two significant results is $$pq^2+(1-p)\alpha^2$$ so the conditional probability you want is $$\frac{pq^2}{pq^2+(1-p)\alpha^2}$$

If you require that $q\geq \alpha$ (so you assume it's impossible for the treatments to be harmful), and take the stereotypical $\alpha=0.025$, the probability is between 0.6 (at $q=0.025$) and 0.984 (at $q=1$).

pf<-function(q,alpha=0.025,fdr=0.4){
    (1-fdr)*alpha/((1-fdr)*alpha+fdr*q)
}

prob2<-function(q,alpha=0.025,fdr=0.4){
    p<-pf(q,alpha,fdr)
    p*q^2/(p*q^2+(1-p)*alpha^2)
}

curve(prob2(x),from=0.025,to=1,ylab="Prob(works)",xlab="Power")