# calculating necessary sample size

I am trying to use python and online tools to calculate the accurate sample size. However, each way I use I get a different result.

This is the data I have from a previous test

Control Group

• Sent:140000
• Converted: 6000
• Conversion Rate: 0.0429

Test/Treatment:

• Sent:350000
• Converted:19000
• Conversion Rate: 0.0543

If I use this calculator http://www.evanmiller.org/ab-testing/sample-size.html with

• Baseline conversion rate 4.29
• minimum detectable effect 2% ( I don't get results for 20%)
• 1 - Beta: 80%
• alpha: 5%
• RESULT: 1,713

If I use this calculator https://www.optimizely.com/resources/sample-size-calculator/?conversion=4.29&effect=20&significance=95 with

• Baseline conversion rate 4.29
• minimum detectable effect: 20%
• statistical significance 95%
• RESULT: 8,300

same calculator

• Baseline conversion rate 4.29
• minimum detectable effect 20% (if I enter 2%I get a sample size required of 1,200,000)
• statistical significance 90%
• RESULT: 7,900

Finally, when I use python (code from here https://stackoverflow.com/questions/15204070/is-there-a-python-scipy-function-to-determine-parameters-needed-to-obtain-a-ta)

from scipy.stats import norm, zscore

def sample_power_probtest(p1, p2, power=0.8, sig=0.05):
z = norm.isf([sig/2]) #two-sided t test
zp = -1 * norm.isf([power])
d = (p1-p2)
s =2*((p1+p2) /2)*(1-((p1+p2) /2))
n = s * ((zp + z)**2) / (d**2)
return int(round(n))

def sample_power_difftest(d, s, power=0.8, sig=0.05):
z = norm.isf([sig/2])
zp = -1 * norm.isf([power])
n = s * ((zp + z)**2) / (d**2)
return int(round(n))

if __name__ == '__main__':
n = sample_power_probtest(0.0429, 0.0543, power=0.8, sig=0.05)
print n


and I get RESULT: 5585

## 3 Answers

You need to enter a 20% relative effect into the first calculator. The result is 9,000 samples. The actual effect is 26% that might be the reason your program returns a different result.

• Welcome to the site! This is more of a comment than an answer, however - we favour in depth answers. This page offers advice on how to answer: stats.stackexchange.com/help/how-to-answer – mkt - Reinstate Monica Mar 31 '18 at 0:32
• Sorry, but my currently low reputation prevents me from leaving comments for the original question :) – iggy Apr 1 '18 at 5:27

For quick calculation, one can use following simplified formula:

sample size = 16 * p * (100-p) / (d ^ 2)


where p = baseline proportion in percent

and d = absolute percent difference

If p=4.29 and d=5.43-4.29=1.14

sample size = 5055


Which is very close to accurate calculations using proper formulae.

Also, if you feed above p and d at https://www.evanmiller.org/ab-testing/sample-size.html

you get sample size of 5,142 which is also close and consistent.

On https://www.optimizely.com/sample-size-calculator/?conversion=4.29&effect=26.6&significance=95 you have feed relative percent difference, i.e. (1.14/4.29)*100 = 26.6%. With these values you get sample size of 4500, which is not close for reasons unclear to me.

This is an old question but it may be useful to add an answer.

Using your values on this reputed online sample size calculator: http://www.sample-size.net/sample-size-proportions/

The standard normal deviate for α = Zα = 1.960
The standard normal deviate for β = Zβ = 0.842
Pooled proportion = P = (q1*P1) + (q0*P0) = 0.049
A = Zα√P(1-P)(1/q1 + 1/q0) = 0.843
B = Zβ√P1(1-P1)(1/q1) + P0(1-P0)(1/q0) = 0.362
C = (P1-P0)2 = 0.000
Total group size = N = (A+B)2/C = 11,168
Continuity correction (added to N for Group 0) = CC = 1/(q1 * |P1-P0|) = 175


Hence 5584 per group.