2
$\begingroup$

Statistical power is defined as the probability of wrongly accepting the null hypothesis for a given sample size, p-value cutoff and effect size.

I am after accepted measures of the converse: the minimum effect size that can be reliably detected given sample size, p-value cutoff and the desired level of reliability of detection.

$\endgroup$

1 Answer 1

1
$\begingroup$

I'll assume that by "level of reliability of detection", you simply mean power.

If so, I don't think there is anything more accepted than "minimum detectable effect size". As in:

By simulation, we found that the minimum effect size (Cohen's $d$) between two groups of size $n=20$ detectable using an unpaired t-test assuming equal variances with $\alpha=0.05$ and power $\beta=0.80$ is $d=0.91$.

Below is code I'd use. I'd change the value for dd until I got the power I wanted. (If you want to be fancy, you can wrap this search around the code.)

nn <- 1e5
kk <- 20
dd <- 0.91
alpha <- 0.05

detect <- rep(FALSE,nn)
for ( ii in 1:nn ) {
    xx <- rnorm(kk)
    yy <- rnorm(kk,dd)
    detect[ii] <- t.test(xx,yy)$p.value<alpha
}

sum(detect)/nn  # change dd until this is close to the desired power
$\endgroup$
2
  • $\begingroup$ This is basically what I have in mind except for Cohen's $d$. I had in mind studies where you compare the probabilities of a (usually rare) event in the two groups. Things like probabilies of ad clicks, deaths, loan defaults, ... So the question is how big a relative or absolute change in probability can you detect for a given $\alpha$, $\beta$ and $n$. Often you have good estimate of the baseline probabilty. $\endgroup$ Apr 7, 2016 at 7:29
  • $\begingroup$ I'd happily adapt a simulation like this to any measure of effect size - (absolute) differences in probabilities or odds, true coefficient values, classification measures like sensitivity or specificity, whatever. Then again, I have never tried putting something like this into a paper submission or project proposal. "Normal" power calculations, where you prespecify the effect size and calculate the necessary sample size, are more common. $\endgroup$ Apr 7, 2016 at 7:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.