0
$\begingroup$

I am slowly getting introduced to A/B testing and holdout testing at my new job. One of the things our A/B testing process stresses on is performing a power analysis before running a test. The common practice I have seen is to use something like the evan miller calculator to perform these power calculations. The one thing I am still not sure I understand is the link between the power analysis and post-experiment analysis. The internal runbook is a power analysis should be performed before running any experiment, but when I read the assumptions behind post experiments tests like t-tests, prop tests etc I don't see any mention of power/effect etc. The three questions that come to my mind are:

  1. Is power analysis a required step before doing an A/B test, holdout? If yes why?
  2. What's the link between a power analysis and hypothesis tests such as a t-test or prop test?
  3. If power analysis is primarily a way to avoid peeking (I am making an assumption here)? Would setting an arbitrary time before the experiment starts to run the analysis on be a sound approach?

I started asking the question because I am not trusting the baseline conversion figures or MDEs that I kept hearing and some of the required sample sizes I am getting are really large and are basically not feasible.

If I have an initial sample of 10000 users and before the start of the experiment I decide to show n% of users variant 1 or no variant, and (1-n)% another variant based on some qualitative assessment let's say, then wouldn't that be a valid experiment?

Thanks.

$\endgroup$
2
  • $\begingroup$ You may also be interested to read Trustworthy Online Controlled Experiments: A Practical Guide to A/B Testing by R. Kohavi, D. Tang and Y. Xu and some of their papers. $\endgroup$
    – dipetkov
    Sep 9, 2022 at 6:43
  • $\begingroup$ Hi @dipetkov, is there a particular chapter that discusses what I am mentioning in the post you can point me to? The book is pretty big and I find some topics are a lot more complex than my current stat level. Any other resources in mind that explain why a power analysis is necessary when most hypothesis tests don't make any required sample size assumptions. $\endgroup$
    – Anas
    Oct 6, 2022 at 6:06

1 Answer 1

0
$\begingroup$

There are at least three points why you should perform a power analysis before the experiment:

  1. If your t-test does not reach statistical significance then you can neither reject the $H_0$ nor the $H_1$. You simply are not any wiser then before the test. Only if you know, that your sample was reasonably sized the non-significant p value has a meaning.

  2. If your t-test is significant, even though your sample size was far to small, something is wrong about your data. There are studies that come up with really unreasonable effects that are not plausible through the experimental group, yet statistically significant (Like in his great talk "Andrew Gelman: Crimes against Data" at minute 7 here). Either (bad) luck or some effect that was not planned mixes with your data. It happens and it is the most likely reason for largely undersized studies to become significant. So you are in a bad spot if the test is significant and you are in a bad spot if the test is not significant.

  3. Your users are probably a worthy ressource and there is a lot of possible things to try and test. (I assume that. My backgroud is medicine and there this point is easier to make.) Should you really claim the next 10,000 users for your experiment and thus block them from any other tests?

It might be valid to just claim that 10,000 users should be large enough for everything -- but if so, it should get you thinking why your power analysis shows the opposite.

All of this is for Frequentist significance testing. The Bayesian in me opposes but citing the t test I assume you really want p values.

$\endgroup$
3
  • $\begingroup$ I have a couple of follow-ups on the above as I am not sure I fully understand your points. $\endgroup$
    – Anas
    Oct 6, 2022 at 5:58
  • $\begingroup$ 1. "If your t-test does not reach statistical significance then you can neither reject the 𝐻0 nor the 𝐻1". Not having statistical significant results would mean we fail to reject the null hypothesis. 2. Again, here I am not sure I understand what the issue is. T-Tests don't have any assumptions around sample sizes. $\endgroup$
    – Anas
    Oct 6, 2022 at 6:04
  • $\begingroup$ Ad 1) when you fail to reject the null hypothesis the next question is usually why. Was it because you did not try hard enough or because the effect just was not there. Ad 2) the test does not make sample size assumptions but if you claimed unreasonable results I will tend not to believe them, regardless of a test result. $\endgroup$
    – Bernhard
    Oct 6, 2022 at 12:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.