calculating necessary sample size

Question

I am trying to use python and online tools to calculate the accurate sample size. However, each way I use I get a different result.

This is the data I have from a previous test

Control Group

• Sent:140000
• Converted: 6000
• Conversion Rate: 0.0429

Test/Treatment:

• Sent:350000
• Converted:19000
• Conversion Rate: 0.0543

If I use this calculator http://www.evanmiller.org/ab-testing/sample-size.html with

• Baseline conversion rate 4.29
• minimum detectable effect 2% ( I don't get results for 20%)
• 1 - Beta: 80%
• alpha: 5%
• RESULT: 1,713

If I use this calculator https://www.optimizely.com/resources/sample-size-calculator/?conversion=4.29&effect=20&significance=95 with

• Baseline conversion rate 4.29
• minimum detectable effect: 20%
• statistical significance 95%
• RESULT: 8,300

same calculator

• Baseline conversion rate 4.29
• minimum detectable effect 20% (if I enter 2%I get a sample size required of 1,200,000)
• statistical significance 90%
• RESULT: 7,900

Finally, when I use python (code from here https://stackoverflow.com/questions/15204070/is-there-a-python-scipy-function-to-determine-parameters-needed-to-obtain-a-ta)

from scipy.stats import norm, zscore

def sample_power_probtest(p1, p2, power=0.8, sig=0.05):
    z = norm.isf([sig/2]) #two-sided t test
    zp = -1 * norm.isf([power]) 
    d = (p1-p2)
    s =2*((p1+p2) /2)*(1-((p1+p2) /2))
    n = s * ((zp + z)**2) / (d**2)
    return int(round(n[0]))

def sample_power_difftest(d, s, power=0.8, sig=0.05):
    z = norm.isf([sig/2])
    zp = -1 * norm.isf([power])
    n = s * ((zp + z)**2) / (d**2)
    return int(round(n[0]))

if __name__ == '__main__':
    n = sample_power_probtest(0.0429, 0.0543, power=0.8, sig=0.05)
    print n

and I get RESULT: 5585

Answer 1

For quick calculation, one can use following simplified formula:

sample size = 16 * p * (100-p) / (d ^ 2)

where p = baseline proportion in percent

and d = absolute percent difference

If p=4.29 and d=5.43-4.29=1.14

sample size = 5055

Which is very close to accurate calculations using proper formulae.

Also, if you feed above p and d at https://www.evanmiller.org/ab-testing/sample-size.html

you get sample size of 5,142 which is also close and consistent.

On https://www.optimizely.com/sample-size-calculator/?conversion=4.29&effect=26.6&significance=95 you have feed relative percent difference, i.e. (1.14/4.29)*100 = 26.6%. With these values you get sample size of 4500, which is not close for reasons unclear to me.

Answer 2

You need to enter a 20% relative effect into the first calculator. The result is 9,000 samples. The actual effect is 26% that might be the reason your program returns a different result.

Answer 3

This is an old question but it may be useful to add an answer.

Using your values on this reputed online sample size calculator: http://www.sample-size.net/sample-size-proportions/

The standard normal deviate for α = Zα = 1.960
The standard normal deviate for β = Zβ = 0.842
Pooled proportion = P = (q1*P1) + (q0*P0) = 0.049
A = Zα√P(1-P)(1/q1 + 1/q0) = 0.843
B = Zβ√P1(1-P1)(1/q1) + P0(1-P0)(1/q0) = 0.362
C = (P1-P0)2 = 0.000
Total group size = N = (A+B)2/C = 11,168
Continuity correction (added to N for Group 0) = CC = 1/(q1 * |P1-P0|) = 175 

Hence 5584 per group.