# Adjusting the estimated effect size mid-experiement - what happens when I do this?

I'm struggling to find a statistically rigorous answer to why it's bad to adjust effect sizes while running an experiment.

Let's say I work for a company and we're doing A/B tests. Some engineer has developed a Cool New Feature for our website and she expects that this will make The Magic Number go up by 0.5%.

We have a tool that, given the base rate of The Magic Number, the number of daily visitors on the page, the desired power and significance, and the expected change, will spit out a runtime for the A/B test. The engineer puts in the numbers, and gets a runtime of 14 days.

After a few days, the engineer checks if the daily number of visitors matches the estimate. She finds that the number of visitors is slightly lower than estimated, so the runtime needs to be adjusted to 17 days. This is all fine, because what we're really computing here is a minimum sample size, and that is not adjusted.

However, and this is the real question, she also finds that at this point, the observed effect size is only 0.25% instead of 0.5%, so she adjusts this too, and the runtime now increases from 17 to 66 days, because otherwise the experiment would be massively underpowered.

Now, of course intuitively this is peeking and you can no longer trust your results. You are observing the effect size prematurely and using that as an estimate. However, I cannot find a paper, book, blog or anything else that explains with any mathematical rigor what exactly happens when you do this.

I don't think this is the same as checking for significance before the experiment is over. I (think I) understand that phenomenon quite well.

Could anyone provide me with an explanation, or link me to some literature?

Thanks a lot.

• Power analysis should not be based on observed effect size--which is what is happening here. For a starting place on the literature see this link: statmodeling.stat.columbia.edu/2018/09/24/… Feb 9, 2023 at 14:07