7
$\begingroup$

Evan Miller has created a well-known online AB test sample size calculator. For the sake of being able to program and modify this formula, I would like to know how to calculate sample size Evan Miller-style by hand.

Personally, I'll calculate such a metric by working backwards from how we calculate a 95% confidence interval with the z-test of proportions around the difference in conversion between the two variations ($\hat{d}$) by setting it zero.

I'll define/assume:

  • $\alpha$ = .05, $\beta$ = .2
  • a 50/50 split between the control and experiment, i.e. $n\_exp$ = $n\_control$
  • the control conversion rate, i.e. the base rate before the experiment = $c$
  • $p$ = pooled conversion rate = (number of exp conversions + number of control conversions/ (n_control + n_experiment)) -> in this context -> $(nc+n(c+\hat{d}))/2n$ = $(2c+\hat{d})/2$

Now time to solve for $n$ ...

$$ \hat{d} + Z_{(1+\alpha)/2} * StandardError = 0 $$ $$ \hat{d} + 1.96 * StandardError = 0 $$ $$ \hat{d} + 1.96 * \sqrt{p(1-p)(\frac{1}{n\_exp} + \frac{1}{n\_control})} = 0$$ $$ \hat{d} + 1.96 * \sqrt{p(1-p)(\frac{2}{n})} = 0$$ $$ \sqrt{p(1-p)(\frac{2}{n})} =\frac{-\hat{d}}{1.96}$$

with more simplifying we get to:

$$ \frac{(1.96^2) 2p(1-p)}{\hat{d}^2} = n $$ $$ \frac{(1.96^2) (2c+2c\hat{d}-2c^2+\frac{3}{2}\hat{d}^2)}{\hat{d}^2} = n $$

At the moment though, my calculation doesn't incorporate power (1-$\beta$) but Evan Miller's does.

What should I think about as next steps to incorporate power into my sample size calculation?

(Feel free to also point out other errors in my calculation or assumptions!)

$\endgroup$
1
$\begingroup$

Have a look at this lecture.

https://newonlinecourses.science.psu.edu/stat414/node/306/

It explains how to calculate sample size for any given level of power in detail. The power is the probability of not making a type 2 error, so it is introduced at the point where you decide on the effect size you want to detect.

$\endgroup$
  • $\begingroup$ Wanna add more details to this answer? $\endgroup$ – zthomas.nc Mar 1 at 17:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.