12
$\begingroup$

Evan Miller has created a well-known online AB test sample size calculator. For the sake of being able to program and modify this formula, I would like to know how to calculate sample size Evan Miller-style by hand.

Personally, I'll calculate such a metric by working backwards from how we calculate a 95% confidence interval with the z-test of proportions around the difference in conversion between the two variations ($\hat{d}$) by setting it zero.

I'll define/assume:

  • $\alpha$ = .05, $\beta$ = .2
  • a 50/50 split between the control and experiment, i.e. $n\_exp$ = $n\_control$
  • the control conversion rate, i.e. the base rate before the experiment = $c$
  • $p$ = pooled conversion rate = (number of exp conversions + number of control conversions/ (n_control + n_experiment)) -> in this context -> $(nc+n(c+\hat{d}))/2n$ = $(2c+\hat{d})/2$

Now time to solve for $n$ ...

$$ \hat{d} + Z_{(1+\alpha)/2} * StandardError = 0 $$ $$ \hat{d} + 1.96 * StandardError = 0 $$ $$ \hat{d} + 1.96 * \sqrt{p(1-p)(\frac{1}{n\_exp} + \frac{1}{n\_control})} = 0$$ $$ \hat{d} + 1.96 * \sqrt{p(1-p)(\frac{2}{n})} = 0$$ $$ \sqrt{p(1-p)(\frac{2}{n})} =\frac{-\hat{d}}{1.96}$$

with more simplifying we get to:

$$ \frac{(1.96^2) 2p(1-p)}{\hat{d}^2} = n $$ $$ \frac{(1.96^2) (2c+2c\hat{d}-2c^2+\frac{3}{2}\hat{d}^2)}{\hat{d}^2} = n $$

At the moment though, my calculation doesn't incorporate power (1-$\beta$) but Evan Miller's does.

What should I think about as next steps to incorporate power into my sample size calculation?

(Feel free to also point out other errors in my calculation or assumptions!)

$\endgroup$
1

2 Answers 2

3
$\begingroup$

try this: $$ n=\frac{(Z_{\alpha/2}\sqrt{2p_1 (1-p_1)}+Z_{\beta}\sqrt{p_1(1-p_1)+p_2(1-p_2)})^2}{|p_2-p_1|^2} $$ where:

  • $p_1$ is the "Baseline conversion rate"

  • $p_2$ is the conversion rate lifted by Absolute "Minimum Detectable Effect", which means $p_1+\text{Absolute Minimum Detectable Effect}$

  • $\alpha$ is the "Significance level $\alpha$"

  • $\beta$ is the $\beta$ in "Statistical power $1−\beta$"

  • $Z_{\alpha/2}$ means Z Score from the z table that corresponds to $\alpha/2$

  • $Z_{\beta}$ means Z Score from the z table that corresponds to $\beta$

in Sample Size Calculator (Evan’s Awesome A/B Tools)

I found the formula in A/B测试系列文章之怎么计算实验所需样本量

when I choose:

  • $p_1=20\%$
  • $p_2=p_1+\text{Absolute Minimum Detectable Effect}=20\%+5\%=25\%$
  • $\alpha = 5\% $
  • $\beta = 20\% $
  • $Z_{\alpha/2}=Z_{5\%/2}=-1.959963985$
  • $Z_{\beta}=Z_{20\%}=-0.841621234$

I got 1030.219283 using this formula, which is 1030 in Size Calculator (Evan’s Awesome A/B Tools)

$\endgroup$
1
  • $\begingroup$ any idea how to derive the formula? $\endgroup$
    – Ryan
    Jun 26, 2021 at 3:50
1
$\begingroup$

I can confirm the chosen answer by looking at the code.

function num_subjects(alpha, power_level, p, delta) {
    if (p > 0.5) {
        p = 1.0 - p;
    }
    var t_alpha2 = ppnd(1.0-alpha/2);
    var t_beta = ppnd(power_level);

    var sd1 = Math.sqrt(2 * p * (1.0 - p));
    var sd2 = Math.sqrt(p * (1.0 - p) + (p + delta) * (1.0 - p - delta));

    return (t_alpha2 * sd1 + t_beta * sd2) * (t_alpha2 * sd1 + t_beta * sd2) / (delta * delta);
}

where ppnd is used to calculate the 𝑍 values.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.