13
$\begingroup$

Evan Miller has created a well-known online AB test sample size calculator. For the sake of being able to program and modify this formula, I would like to know how to calculate sample size Evan Miller-style by hand.

Personally, I'll calculate such a metric by working backwards from how we calculate a 95% confidence interval with the z-test of proportions around the difference in conversion between the two variations ($\hat{d}$) by setting it zero.

I'll define/assume:

  • $\alpha$ = .05, $\beta$ = .2
  • a 50/50 split between the control and experiment, i.e. $n\_exp$ = $n\_control$
  • the control conversion rate, i.e. the base rate before the experiment = $c$
  • $p$ = pooled conversion rate = (number of exp conversions + number of control conversions/ (n_control + n_experiment)) -> in this context -> $(nc+n(c+\hat{d}))/2n$ = $(2c+\hat{d})/2$

Now time to solve for $n$ ...

$$ \hat{d} + Z_{(1+\alpha)/2} * StandardError = 0 $$ $$ \hat{d} + 1.96 * StandardError = 0 $$ $$ \hat{d} + 1.96 * \sqrt{p(1-p)(\frac{1}{n\_exp} + \frac{1}{n\_control})} = 0$$ $$ \hat{d} + 1.96 * \sqrt{p(1-p)(\frac{2}{n})} = 0$$ $$ \sqrt{p(1-p)(\frac{2}{n})} =\frac{-\hat{d}}{1.96}$$

with more simplifying we get to:

$$ \frac{(1.96^2) 2p(1-p)}{\hat{d}^2} = n $$ $$ \frac{(1.96^2) (2c+2c\hat{d}-2c^2+\frac{3}{2}\hat{d}^2)}{\hat{d}^2} = n $$

At the moment though, my calculation doesn't incorporate power (1-$\beta$) but Evan Miller's does.

What should I think about as next steps to incorporate power into my sample size calculation?

(Feel free to also point out other errors in my calculation or assumptions!)

$\endgroup$
1

3 Answers 3

3
$\begingroup$

try this: $$ n=\frac{(Z_{\alpha/2}\sqrt{2p_1 (1-p_1)}+Z_{\beta}\sqrt{p_1(1-p_1)+p_2(1-p_2)})^2}{|p_2-p_1|^2} $$ where:

  • $p_1$ is the "Baseline conversion rate"

  • $p_2$ is the conversion rate lifted by Absolute "Minimum Detectable Effect", which means $p_1+\text{Absolute Minimum Detectable Effect}$

  • $\alpha$ is the "Significance level $\alpha$"

  • $\beta$ is the $\beta$ in "Statistical power $1−\beta$"

  • $Z_{\alpha/2}$ means Z Score from the z table that corresponds to $\alpha/2$

  • $Z_{\beta}$ means Z Score from the z table that corresponds to $\beta$

in Sample Size Calculator (Evan’s Awesome A/B Tools)

I found the formula in A/B测试系列文章之怎么计算实验所需样本量

when I choose:

  • $p_1=20\%$
  • $p_2=p_1+\text{Absolute Minimum Detectable Effect}=20\%+5\%=25\%$
  • $\alpha = 5\% $
  • $\beta = 20\% $
  • $Z_{\alpha/2}=Z_{5\%/2}=-1.959963985$
  • $Z_{\beta}=Z_{20\%}=-0.841621234$

I got 1030.219283 using this formula, which is 1030 in Size Calculator (Evan’s Awesome A/B Tools)

$\endgroup$
1
  • $\begingroup$ any idea how to derive the formula? $\endgroup$
    – Ryan
    Jun 26, 2021 at 3:50
2
$\begingroup$

I can confirm the chosen answer by looking at the code.

function num_subjects(alpha, power_level, p, delta) {
    if (p > 0.5) {
        p = 1.0 - p;
    }
    var t_alpha2 = ppnd(1.0-alpha/2);
    var t_beta = ppnd(power_level);

    var sd1 = Math.sqrt(2 * p * (1.0 - p));
    var sd2 = Math.sqrt(p * (1.0 - p) + (p + delta) * (1.0 - p - delta));

    return (t_alpha2 * sd1 + t_beta * sd2) * (t_alpha2 * sd1 + t_beta * sd2) / (delta * delta);
}

where ppnd is used to calculate the 𝑍 values.

$\endgroup$
0
$\begingroup$

This even simpler code gets me the exact same results as Evan Miller's calculator. It's an N-1 variant of the two proportion test (normally the N-1 is used for chi squared tests but it's mathematically equivalent anyway):

n = 2z^2 p (1 - p) / d^2 + 1/2

z^2 is the sum of z for alpha + beta.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.