I consider an A/B test where there is a control group and a variant group. Each observation can either be true (converted) or false (not converted). I evenly and randomly split the incoming users to the two treatments. So, the results can be summarized in a contingency table:
| | Converted | Not converted | |-------|-----------|---------------| |Control| | | |Variant| | |
Let the conversion rate be
Converted / (Converted + Not Converted).
The null hypothesis is that the conversion rate is independent of the treatment.
It seems like in this case, I can use either the two-tailed $z$-test or the $\chi^2$-test. Feel free to correct me on this one.
Determining the sample size
I want to use
import statsmodels.stats.power as power zpower = power.NormalIndPower() chipower = power.GofChisquarePower() zpower.solve_power(0.1, nobs1=None, alpha=0.05, power=0.9, ratio=1.) # Returns ~2100 chipower.solve_power(0.1, nobs=None, alpha=0.05, power=0.9) # Returns ~1050
Question: I am puzzled by the huge difference. What is the reason for it? Am I using something wrongly in regards to my assumptions?
N.B. I now realize that the documentation states that
(solves) for any one parameter of the power of a one sample chisquare-test
(solves) for any one parameter of the power of a two sample z-test
What is the difference between the one sample and two samples?