0
$\begingroup$

I'm struggling to find a statistically rigorous answer to why it's bad to adjust effect sizes while running an experiment.

Let's say I work for a company and we're doing A/B tests. Some engineer has developed a Cool New Feature for our website and she expects that this will make The Magic Number go up by 0.5%.

We have a tool that, given the base rate of The Magic Number, the number of daily visitors on the page, the desired power and significance, and the expected change, will spit out a runtime for the A/B test. The engineer puts in the numbers, and gets a runtime of 14 days.

After a few days, the engineer checks if the daily number of visitors matches the estimate. She finds that the number of visitors is slightly lower than estimated, so the runtime needs to be adjusted to 17 days. This is all fine, because what we're really computing here is a minimum sample size, and that is not adjusted.

However, and this is the real question, she also finds that at this point, the observed effect size is only 0.25% instead of 0.5%, so she adjusts this too, and the runtime now increases from 17 to 66 days, because otherwise the experiment would be massively underpowered.

Now, of course intuitively this is peeking and you can no longer trust your results. You are observing the effect size prematurely and using that as an estimate. However, I cannot find a paper, book, blog or anything else that explains with any mathematical rigor what exactly happens when you do this.

I don't think this is the same as checking for significance before the experiment is over. I (think I) understand that phenomenon quite well.

Could anyone provide me with an explanation, or link me to some literature?

Thanks a lot.

$\endgroup$
1

1 Answer 1

0
$\begingroup$

What you're describing is sometimes described as a sequential analysis. Generally, it's only valid if you plan when you're going to check in advance and then apply the appropriate corrections.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.