I run marketing experiments on a website.

As far as I know, it's a well-known rule that we should estimate required sample size to detect a certain uplift in the treatment before running the experiment. But here's a problem that I encountered.

I estimated the required sample size based on the base conversion rate of the previous month and it was around 9%, but after finishing the experiment, we discovered that the base conversion rate during the period the experiment was performed dropped to 6.7%. The difference was too big to assume that it was simply a sampling error.

Our website traffic is subject to seasonality and we have different conversion rates for different months, days of week etc. Besides that, the quality of traffic we get also changes heavily all the time, so the base conversion rate always heavily fluctuates, which makes it difficult to know the "true" base conversion rate to properly estimate the required sample size.

Is there a way to account for these fluctuations when calculating required sample size? My first though was to "underestimate" the base conversion rate when calculating the sample size so that we still have a sufficient power even if the real base conversion rate drops during the experiment run.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.