Why are statistical power calculations required before A/B tests?

Question

I am slowly getting introduced to A/B testing and holdout testing at my new job. One of the things our A/B testing process stresses on is performing a power analysis before running a test. The common practice I have seen is to use something like the evan miller calculator to perform these power calculations. The one thing I am still not sure I understand is the link between the power analysis and post-experiment analysis. The internal runbook is a power analysis should be performed before running any experiment, but when I read the assumptions behind post experiments tests like t-tests, prop tests etc I don't see any mention of power/effect etc. The three questions that come to my mind are:

1. Is power analysis a required step before doing an A/B test, holdout? If yes why?
2. What's the link between a power analysis and hypothesis tests such as a t-test or prop test?
3. If power analysis is primarily a way to avoid peeking (I am making an assumption here)? Would setting an arbitrary time before the experiment starts to run the analysis on be a sound approach?

I started asking the question because I am not trusting the baseline conversion figures or MDEs that I kept hearing and some of the required sample sizes I am getting are really large and are basically not feasible.

If I have an initial sample of 10000 users and before the start of the experiment I decide to show n% of users variant 1 or no variant, and (1-n)% another variant based on some qualitative assessment let's say, then wouldn't that be a valid experiment?

Thanks.

Answer 1

There are at least three points why you should perform a power analysis before the experiment:

1. If your t-test does not reach statistical significance then you can neither reject the $H_0$ nor the $H_1$. You simply are not any wiser then before the test. Only if you know, that your sample was reasonably sized the non-significant p value has a meaning.

2. If your t-test is significant, even though your sample size was far to small, something is wrong about your data. There are studies that come up with really unreasonable effects that are not plausible through the experimental group, yet statistically significant (Like in his great talk "Andrew Gelman: Crimes against Data" at minute 7 here). Either (bad) luck or some effect that was not planned mixes with your data. It happens and it is the most likely reason for largely undersized studies to become significant. So you are in a bad spot if the test is significant and you are in a bad spot if the test is not significant.

3. Your users are probably a worthy ressource and there is a lot of possible things to try and test. (I assume that. My backgroud is medicine and there this point is easier to make.) Should you really claim the next 10,000 users for your experiment and thus block them from any other tests?

It might be valid to just claim that 10,000 users should be large enough for everything -- but if so, it should get you thinking why your power analysis shows the opposite.

All of this is for Frequentist significance testing. The Bayesian in me opposes but citing the t test I assume you really want p values.