We are designing a a/b test at my company. We have a given possible total sample size of 20,000 and even though a 50/50 split of the control and treatment group has the highest statistical power, the business requirement asks for a smaller treatment group.
I understand that after we run test, we can do a t-test or worst scenario a power analysis.
But before we run the test, I need to establish the smallest size of the treatment group while maintaining statistical significance. (and I need to convince my colleagues)
I was going to do the following:
I used this calculator to establish the minimum sample size http://www.evanmiller.org/ab-testing/sample-size.html
I could also use python, but I want the business to be able to come to the same conclusion without understanding python.
Baseline conversion rate: 10% (based on previous year's conversion rate, since this is the first test and we don't have past test's conversions)
Minimum detectable effect: 2%
Significance level α: 5% [95% confidence interval]
For Statistical power 1−β: 95% --> I got the necessary sample size of : 8,484
For Statistical power 1−β: 80% --> I got the necessary sample size of : 5,083
So this would be my argument, to have the treatment group be at least 8,484. (Making the control group - 11,516)
Is there a better way to convince business, before the test runs and we have actual variances? I am pretty sure that they will argue for the 5,083 group for now. Is the best way to just run the test and then do a t-test and adjust the sizing afterwards? I don't feel comfortable with that, but I think until we have actual results to base our future tests on, I won't have a strong enough argument.
UPDATE: would this be a good approach? https://d17h27t6h515a5.cloudfront.net/topher/2016/December/5845e980_empirical-sizing/empirical-sizing.r