0
$\begingroup$

I have only come across A/B test calculators that give the minimum sample size per variation for 50-50 split between control and treatment.

I am running an A/B test for conversion rates with a 80/20 split between control and treatment. I need to calculate the minimum sample size needed for statistical significance. Can anyone point me to the calculator that does so for uneven split between control and treatment?

Example calculator i have found so far :'https://www.evanmiller.org/ab-testing/sample-size.html' . This calculator is only for 50-50 split.

| cite | improve this question | |
$\endgroup$
  • $\begingroup$ Please describe what about the problem made a hypothesis test (test for existence of a signal) appropriate, as opposed to estimating the magnitude of an effect (with sample size chosen to achieve a certain precision in the estimate, i.e., margin of error). $\endgroup$ – Frank Harrell Jun 30 '19 at 12:40
0
$\begingroup$

From the point of view of the 'power' of the test, the most efficient design is to have equal sample sizes in A and B, so it is customary for software programs and online calculators to assume that $n_1 = n_2$ and then give you the number required in each group to achieve a given power for detecting a given difference in means.

In your case, I suggest you take the answer to be the size of the treatment group, because the power will depend on that more crucially than on the size of the larger control group. ("A chain is only as strong as it's weakest link.")

If you could say what kind of test your will do, it might be possible to give a formula based on both $n_1$ and $n_2.$ If not a formula, then a simulation method.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

Compute the Cohen's D: click here

According to the power you want about your test, here the table that gives you everything: https://en.wikipedia.org/wiki/Sample_size_determination#Estimating_sample_sizes

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.