What A/B test calculator can I use to calculate the minimum same size for uneven split between control and treatment groups?

Question

I have only come across A/B test calculators that give the minimum sample size per variation for 50-50 split between control and treatment.

I am running an A/B test for conversion rates with a 80/20 split between control and treatment. I need to calculate the minimum sample size needed for statistical significance. Can anyone point me to the calculator that does so for uneven split between control and treatment?

Example calculator i have found so far :'https://www.evanmiller.org/ab-testing/sample-size.html' . This calculator is only for 50-50 split.

Answer 1

From the point of view of the 'power' of the test, the most efficient design is to have equal sample sizes in A and B, so it is customary for software programs and online calculators to assume that $n_1 = n_2$ and then give you the number required in each group to achieve a given power for detecting a given difference in means.

In your case, I suggest you take the answer to be the size of the treatment group, because the power will depend on that more crucially than on the size of the larger control group. ("A chain is only as strong as it's weakest link.")

If you could say what kind of test your will do, it might be possible to give a formula based on both $n_1$ and $n_2.$ If not a formula, then a simulation method.

Answer 2

Compute the Cohen's D: click here

According to the power you want about your test, here the table that gives you everything: https://en.wikipedia.org/wiki/Sample_size_determination#Estimating_sample_sizes

Answer 3

Good question, answered by Lachin in his book Biostatistical Methods.

Assuming your AB test is in terms of a binary outcome, the sample size formula is

$$ N = \Big( \dfrac{Z_{1-\alpha} \phi_0 + z_{1-\beta}\phi_1} {\Delta} \Big)^2$$

Here,

• $\Delta$ is the expected difference between groups.
• $\phi_0 = \pi (1-\pi)\times 6.25$ is the variance component under the null. $\pi$ is the expected success rate under the null.
• $\phi_1 = 5\times\pi_1(1-\pi_1) + 1.25\times \pi_2(1-\pi_2)$ is the variance component under the alternative. $\pi_1$ is the expected success rate in the 20% exposure, $\pi_2$ is the expected success rate in the 80% exposure group.

I'm not aware of any calculators which implement this, but this is easily done in R or excel, or even pen and paper frankly.