Assumptions
I consider an A/B test where there is a control group and a variant group. Each observation can either be true (converted) or false (not converted). I evenly and randomly split the incoming users to the two treatments. So, the results can be summarized in a contingency table:
| | Converted | Not converted |
|-------|-----------|---------------|
|Control| | |
|Variant| | |
Let the conversion rate be Converted / (Converted + Not Converted)
.
The null hypothesis is that the conversion rate is independent of the treatment.
It seems like in this case, I can use either the two-tailed $z$-test or the $\chi^2$-test. Feel free to correct me on this one.
Determining the sample size
I want to use statsmodels.stats.power.GofChisquarePower.solve_power
and statsmodels.stats.power.NormalIndPower.solve_power
.
For example:
import statsmodels.stats.power as power
zpower = power.NormalIndPower()
chipower = power.GofChisquarePower()
zpower.solve_power(0.1, nobs1=None, alpha=0.05, power=0.9, ratio=1.) # Returns ~2100
chipower.solve_power(0.1, nobs=None, alpha=0.05, power=0.9) # Returns ~1050
Question: I am puzzled by the huge difference. What is the reason for it? Am I using something wrongly in regards to my assumptions?
N.B. I now realize that the documentation states that GofChisquarePower.solve_power
(solves) for any one parameter of the power of a one sample chisquare-test
and NormalIndPower.solve_power
(solves) for any one parameter of the power of a two sample z-test
What is the difference between the one sample and two samples?