3
$\begingroup$

I am trying to use python and online tools to calculate the accurate sample size. However, each way I use I get a different result.

This is the data I have from a previous test

Control Group

  • Sent:140000
  • Converted: 6000
  • Conversion Rate: 0.0429

Test/Treatment:

  • Sent:350000
  • Converted:19000
  • Conversion Rate: 0.0543

If I use this calculator http://www.evanmiller.org/ab-testing/sample-size.html with

  • Baseline conversion rate 4.29
  • minimum detectable effect 2% ( I don't get results for 20%)
  • 1 - Beta: 80%
  • alpha: 5%
  • RESULT: 1,713

If I use this calculator https://www.optimizely.com/resources/sample-size-calculator/?conversion=4.29&effect=20&significance=95 with

  • Baseline conversion rate 4.29
  • minimum detectable effect: 20%
  • statistical significance 95%
  • RESULT: 8,300

same calculator

  • Baseline conversion rate 4.29
  • minimum detectable effect 20% (if I enter 2%I get a sample size required of 1,200,000)
  • statistical significance 90%
  • RESULT: 7,900

Finally, when I use python (code from here https://stackoverflow.com/questions/15204070/is-there-a-python-scipy-function-to-determine-parameters-needed-to-obtain-a-ta)

from scipy.stats import norm, zscore

def sample_power_probtest(p1, p2, power=0.8, sig=0.05):
    z = norm.isf([sig/2]) #two-sided t test
    zp = -1 * norm.isf([power]) 
    d = (p1-p2)
    s =2*((p1+p2) /2)*(1-((p1+p2) /2))
    n = s * ((zp + z)**2) / (d**2)
    return int(round(n[0]))

def sample_power_difftest(d, s, power=0.8, sig=0.05):
    z = norm.isf([sig/2])
    zp = -1 * norm.isf([power])
    n = s * ((zp + z)**2) / (d**2)
    return int(round(n[0]))

if __name__ == '__main__':
    n = sample_power_probtest(0.0429, 0.0543, power=0.8, sig=0.05)
    print n

and I get RESULT: 5585

$\endgroup$
1
$\begingroup$

You need to enter a 20% relative effect into the first calculator. The result is 9,000 samples. The actual effect is 26% that might be the reason your program returns a different result.

$\endgroup$
  • $\begingroup$ Welcome to the site! This is more of a comment than an answer, however - we favour in depth answers. This page offers advice on how to answer: stats.stackexchange.com/help/how-to-answer $\endgroup$ – mkt - Reinstate Monica Mar 31 '18 at 0:32
  • $\begingroup$ Sorry, but my currently low reputation prevents me from leaving comments for the original question :) $\endgroup$ – iggy Apr 1 '18 at 5:27
0
$\begingroup$

This is an old question but it may be useful to add an answer.

Using your values on this reputed online sample size calculator: http://www.sample-size.net/sample-size-proportions/

The standard normal deviate for α = Zα = 1.960
The standard normal deviate for β = Zβ = 0.842
Pooled proportion = P = (q1*P1) + (q0*P0) = 0.049
A = Zα√P(1-P)(1/q1 + 1/q0) = 0.843
B = Zβ√P1(1-P1)(1/q1) + P0(1-P0)(1/q0) = 0.362
C = (P1-P0)2 = 0.000
Total group size = N = (A+B)2/C = 11,168
Continuity correction (added to N for Group 0) = CC = 1/(q1 * |P1-P0|) = 175 

Hence 5584 per group.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.