Before starting an a/b test with a control and one experimental route, I can calculate a required sample size based on conversion rate estimates for both routes. I can get a good estimate of the conversion rate of the control by looking at historical data. But the conversion rate of the experimental route is unknown. What I would like to do is calculate a number of different sample sizes based on a variety of sensitivities.
For example, I can calculate sample sizes for a 10%, 15% and 20% sensitivity (increase in conversion from the control) that might look like this:
Sensitivity Required Sample Size 10% 1,961 15% 871 20% 490
Some of the reading I've done says that you should calculate a single sample size at the start of the test and always run the test for that long.
- Is there any problem with checking for statistical significance at multiple pre-calculated sample sizes and potentially ending a test early if I've found the results to be statistically significant?
I originally estimate that the experimental route will outperform the control by 15%. But once I've reached 490 samples I find that the experimental route is actually outperforming the control by 20%, can I end the test and declare that the experimental route boosts conversion by 20%?