Business requirement for small sample size impacts statistical power

Question

We are designing a a/b test at my company. We have a given possible total sample size of 20,000 and even though a 50/50 split of the control and treatment group has the highest statistical power, the business requirement asks for a smaller treatment group.

I understand that after we run test, we can do a t-test or worst scenario a power analysis.

But before we run the test, I need to establish the smallest size of the treatment group while maintaining statistical significance. (and I need to convince my colleagues)

I was going to do the following:

I used this calculator to establish the minimum sample size http://www.evanmiller.org/ab-testing/sample-size.html

I could also use python, but I want the business to be able to come to the same conclusion without understanding python.

Baseline conversion rate: 10% (based on previous year's conversion rate, since this is the first test and we don't have past test's conversions)

Minimum detectable effect: 2%

Significance level α: 5% [95% confidence interval]

For Statistical power 1−β: 95% --> I got the necessary sample size of : 8,484

For Statistical power 1−β: 80% --> I got the necessary sample size of : 5,083

So this would be my argument, to have the treatment group be at least 8,484. (Making the control group - 11,516)

Is there a better way to convince business, before the test runs and we have actual variances? I am pretty sure that they will argue for the 5,083 group for now. Is the best way to just run the test and then do a t-test and adjust the sizing afterwards? I don't feel comfortable with that, but I think until we have actual results to base our future tests on, I won't have a strong enough argument.

UPDATE: would this be a good approach? https://d17h27t6h515a5.cloudfront.net/topher/2016/December/5845e980_empirical-sizing/empirical-sizing.r

Answer 1

According to Glen_b's answer here the only important value related to sample sizes is $\sqrt{\frac{1}{n_1}+\frac{1}{n_2}}$.

This is the important value for the T statistic:

$t=\frac{\bar{x_1}-\bar{x_2}}{\sigma \sqrt{\frac{1}{n_1}+\frac{1}{n_2}}}$

The power of the test should be equal when $\sqrt{\frac{1}{n_1}+\frac{1}{n_2}}$ is equal. If you use this fact then you can get the value of $\sqrt{\frac{1}{n_1}+\frac{1}{n_2}}$ for your chosen sample sizes and get the power of a test with equal sample sizes which gives the same value of $\sqrt{\frac{1}{n_1}+\frac{1}{n_2}}$

edit:

If you don't like the above approach you can do your own power analysis. Rather than computing the power exactly you can find it by a simulation.

The power $1-\beta$ is the chance your test will find that the null hypothesis is false given that it is in fact false.

You've chosen a baseline conversion rate of 10% and minimum detectable effect of 2% so you can choose the control group to have a mean of 10% and the treatment group to have a mean of 12% (This makes slightly more variance than means of 8% and 10%).

With your chosen sample sizes use a computer to generate random samples for both the control and treatment groups, then conduct a t-test and record if the null hypothesis was rejected or not. Repeat this many times to simulate the experiment over and over, from this you can see what the chance of rejecting the null hypothesis is and that is the power of your test.

This approach is a bit more rigorous than the first method I mentioned for estimating the power.