0
$\begingroup$

I am trying to find a proper sample size for a project I'm working on and just realized that the values I have calculated are significantly different from an online calculator (which is supposed to be the same).

If I calculate the sample size in this online calculator, the result is 1.6 million, whereas if I use the same parameters in R using a power analysis of two independent proportions, the estimated sample size I get is around 1,130:

power.prop.test(p1=0.15, p2=0.10, power=0.95, alternative='two.sided', sig.level=0.05)

Any ideas on why there is such a difference?

Improve this question
$\endgroup$
6
  • $\begingroup$ I don't have time to check the documentation for the R function you've used, but I suspect that while the "p1" argument corresponds to the "baseline conversion rate", "p2" does not correspond to the "minimum detectable effect". Rather, the minimum detectable effect may be the difference between p1 and p2 (or some measure of effect size) $\endgroup$
    – Ian_Fin
    Oct 25, 2016 at 14:42
  • $\begingroup$ In addition to that, the website doesn't say what the beta level it's using is (i.e. "power"), so can't be sure that it's .95, as is the case in that R function. $\endgroup$
    – Ian_Fin
    Oct 25, 2016 at 14:44
  • 1
    $\begingroup$ The link you included also has a FAQ with an answer to the quetsion: "Why is your calculator different from other sample size calculators?". It doesn't answer your question completely I think. But my perspective: there are different ways to compute sample sizes for different types of tests. Clearly, these computations have very different assumption.s $\endgroup$
    – Ivo
    Oct 25, 2016 at 14:49
  • $\begingroup$ It looks like you specified the effect to the web calculator as 10. Is that what you meant to do? $\endgroup$
    – mdewey
    Oct 25, 2016 at 15:13
  • 2
    $\begingroup$ You also used 0.15% in the calculator and 15% in the R function. Mind your decimals. $\endgroup$
    – Pierre L
    Oct 25, 2016 at 16:28

1 Answer 1

Reset to default
1
$\begingroup$

The second parameter is the second proportion. So you need to define according to the MDE used in the online calculator. As other said, there is not information on the power.

With the following parameters we get same approximation

power.prop.test(p1=0.15, p2=0.15*(1-0.10), power=0.80, alternative='two.sided', sig.level=0.05)

     Two-sample comparison of proportions power calculation 

              n = 8524.013
             p1 = 0.15
             p2 = 0.135
      sig.level = 0.05
          power = 0.8
    alternative = two.sided

NOTE: n is number in *each* group

See this for more information https://signalvnoise.com/posts/3004-ab-testing-tech-note-determining-sample-size

Improve this answer
$\endgroup$
1
  • $\begingroup$ This together with Pierre Lafortune's comment made me realize my issue. The sample size is indeed very big. $\endgroup$
    – johan855
    Oct 26, 2016 at 9:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.