Where to read about how to build a sample size calculator for power analysis for a/b tests

Question

I am wondering how to build a sample size calculator like this one. Will the intro chapters of a stats textbook cover this? I've taken a couple stats classes but I'm not sure how baseline effects and minimum detectable effects come into this.

Answer 1

Most books should cover some sample size derivations but may not cover this one. The general idea is to use the confidence interval but solve for $n$ instead by defining all the other variables. So the baseline effect would be the center of the confidence interval and the minimum detectable effect is the width of the interval. That site is using the formula:

$$ \begin{align} p_1&=baseline\\ p_2&=baseline + absoluteMinimumDetectableEffect\\ \sqrt{n} &= \frac{\sqrt{2p_1(1-p_1)}z_{1-\alpha/2} + \sqrt{p_1(1-p_1) + p_2(1-p_2)}z_{1-\beta}}{|p_1-p_2|} \end{align} $$

This formula includes $power=1-\beta$ so the derivation actually comes from:

$$ P\Bigg\{\frac{|p_2-p_1|}{\sqrt{2p_1(1-p_1)/n}}>z_{\alpha/2} \Bigg\} = 1 - \beta $$

This other site has a calculator with a slightly different formula credited from Bernard Rosner's Fundamentals of Biostatistics. Instead of asking for a baseline and minimum detectable effect, it asks for $p_1$ and $p_2$ directly.

$$ \begin{align} p_{avg}&=\frac{1}{2}(p_1 + p_2)\\ \sqrt{n} &= \frac{\sqrt{2p_{avg}(1-p_{avg})}z_{1-\alpha/2} + \sqrt{p_1(1-p_1) + p_2(1-p_2)}z_{1-\beta}}{|p_1-p_2|} \end{align} $$

This third site adds a continuity correction credited from Statistical Methods for Rates and Proportions by Joseph L. Fleiss. Given the second defintion of $n$ above:

$$ n^*=\frac{n}{4}\Bigg(1 + \sqrt{1+\frac{4}{n|p_1-p_2|}}\Bigg)^2 $$