3
$\begingroup$

This is probably basic, but I haven't come across it before:

With a minimum detectable effect of 10% and a baseline conversion rate of 0.44% it takes 265,857 observations per cohort to reach 80% power (1-sided test with 5% alpha).

Keeping the same parameters, but changing the baseline rate to, say, 44%, we see only 1,573 observations are needed per cohort.

Why does the value of the conversion rate change the number of observations required?

Code (Using R power.prop.test)

# Choose baseline (control) conversion rate
      #BaselineConversion <- 0.0044 # <- This is the real conversion rate
      BaselineConversion <-  0.44   # <- This is the adjusted rate for comparision
    
  # We want to be able to detect a minumium of 10% drop in conversion rates (i.e. if the reduction in conversion rates is <10% we don't care)
    minDetectedDrop_10pct <- 0.1
    
  # Power calculation: -10%
    minDetectedConversion_10pct <- BaselineConversion*(1-minDetectedDrop_10pct)
    testResult_10pct <- power.prop.test(
      p1= BaselineConversion,
      p2 = minDetectedConversion_10pct,
      sig.level = 0.05,
      power = 0.8,
      alternative = 'one.sided')
    paste0('Number observations needed with baseline conversion of: ',BaselineConversion,' is: ', round(testResult_10pct$n))
$\endgroup$
3
  • 1
    $\begingroup$ The bigger the effect size, the less observation you will need to show a statistical significance (got I hate that phrase). $\endgroup$ Commented Jul 24, 2020 at 8:52
  • $\begingroup$ Thanks for your reply... so I guess when you say 'big' you mean relative to the denominator right? So even though the numerator could be hundreds of thousands, if the denominator is much bigger, that's still a small effect size. $\endgroup$
    – SeánMcK
    Commented Jul 24, 2020 at 14:15
  • $\begingroup$ If you measure A LOT of giraffes in Africa and you notice a 1 mm statistically significant difference in height between the Masai and Angolian population... would you make this biologically significant? Hint, giraffes can grow up to 5-6 meters. $\endgroup$ Commented Jul 27, 2020 at 9:35

1 Answer 1

0
$\begingroup$

The average number of success with success rate p is p for binomial distribution. So the difference between p and 0.9 * p is proportional to p. In your examples, 10% of 0.44 is much greater than 10% of 0.0044 .

The figure below shows the probability density of the success number from 1000 draws. The success rates are 0.05, 0.5, 0.95 for the solid lines from left to right. The success rates of the dashed lines are 10% smaller than the solid lines.

enter image description here

Observe that the variance is wider in the middle and thinner on both sides, but its effect on power is not as strong as the distance between peaks. (Note that the variance of average success is p(1-p)/n)

The R code for the plot.

N = 1000
x_ = 0:N
plot(x_/N, dbinom(x_, size=N, prob=0.05) * N, type='l', xlab='Mean success number', ylab='Density')
lines(x_/N, dbinom(x_, size=N, prob=0.05*0.9) * N, lty=2)

lines(x_/N, dbinom(x_, size=N, prob=0.5) * N, col='red')
lines(x_/N, dbinom(x_, size=N, prob=0.5*0.9) * N, lty=2, col='red')

lines(x_/N, dbinom(x_, size=N, prob=0.95) * N, col='blue')
lines(x_/N, dbinom(x_, size=N, prob=0.95*0.9) * N, lty=2, col='blue')
$\endgroup$
4
  • $\begingroup$ Thanks for the clear answer (and sorry for the delay in accepting). I think this was a side remark, but I'm curious why is the 'variance in the middle'? $\endgroup$
    – SeánMcK
    Commented Sep 1, 2020 at 8:23
  • $\begingroup$ Do you mean why is the variance in the middle wider? The variance of binomial distribution is np(1-p). So the variance of binomial distribution divided by n (i.e. success rate) is p(1-p)/n. if p=0.1, 0.1*0.9 = 0.09; if p=0.5, 0.5*0.5 = 0.25; if p=0.9, 0.9*0.1 = 0.09; $\endgroup$ Commented Sep 1, 2020 at 9:15
  • $\begingroup$ Ok cool, so it's just a mechanical effect from the definition of variance in a binomial distribution. Thanks for all the clarity! $\endgroup$
    – SeánMcK
    Commented Sep 1, 2020 at 10:31
  • $\begingroup$ Thank you for the interesting question! After thinking twice, both extreme cases do give a feeling of less "surprises" than the middle 50/50 case does. $\endgroup$ Commented Sep 1, 2020 at 12:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.