1
$\begingroup$

I was trying to understand the math behind some commonly used calculators and formulas for A/B tests and it seems that there might be some variations. Ideally, I would like to understand how each of them is derived and under what assumptions.

$$\left(\frac{\Phi_{1-\alpha/2}^{-1}\sqrt{2\bar p(1-\bar p)}+\Phi_{1 -\beta}^{-1}\sqrt{p_1(1-p_1)+p_2(1-p_2)}}{|p_2-p_1|}\right)^2$$ where $$ \bar p = \frac{p_1 + p_2}{2}$$ as far as I understand.

This is the formula I found here, for example: https://jeffshow.com/caculate-abtest-required-sample-size.html

It's in Chinese, but from what I read after using Google Translate, this is exactly that type of calculation.

Honestly speaking, I have no idea where this formula comes from.

In the general case, the formula for the sample size is:

$$n \ge 2 \Big(\frac{\Phi^{-1}(1-\alpha/2)+\Phi^{-1}(1-\beta)}{\Delta/\sigma}\Big)^2$$

Under the assumption of equal variances, we can calculate the pooled variance as: $$\sigma_p=\sqrt{\dfrac{(n_1-1)s_1^2+(n_2-1)s^2_2}{n_1+n_2-2}}$$, which can be simplified given that $n_1 = n_2$ as $\sigma_p = \sqrt{\frac{p_1\cdot(1-p_1) + p_2 \cdot (1-p_2)}{2}}$.

Plugging that into the formula for the general case, we get:

$$n \geq \frac{p_1\cdot(1-p_1) + p_2 \cdot (1-p_2)}{(p_2-p_1)^2} \cdot (z_{\frac{\alpha}{2}} + z_{\beta})^2$$

The results, however, are not the same when using these two formulas. As far as I understand, the famous Evan Miller calculator is based on the first formula.

For $p_1=0.5$ and $p_2=0.6$, my formula gives $n=384.16$ whereas this calculator reports $390$ as the answer.

Can you please clarify it a bit? I believe there might be some assumptions or, even worse, something really trivial I fail to take into consideration.

Thanks!

$\endgroup$
1
  • 1
    $\begingroup$ You were so close! :) (+1) $\endgroup$ – usεr11852 Apr 18 at 3:25
1
+50
$\begingroup$

The Maths is correct; the interpretation of what is the baseline needs a bit adjusting to get back to the EM result.

When comparing two arbitrary proportion coming from equal sized population, indeed the base-line proportion is $\bar{p}$ is $\frac{p_1 + p_2}{2}$. When though making an A/B test the base-line proportion is $p_1$. As such if we use $\bar{p}=p_1$ then we can get the results in question:

p1 = 0.5
p2 = 0.6
delta = abs(p1-p2)
alpha_level = 0.05
beta_level = 0.80
z_alpha = qnorm(1.0-alpha_level/2)
z_beta = qnorm(beta_level)

sd1_ab = sqrt(2*p1*(1.0-p1))
sd2 = sqrt(p1*(1-p1)+p2*(1-p2))
((z_alpha * sd1_ab + z_beta * sd2) / abs(delta))^2 # 390.0778 as expected

Now, what you have done though is to correctly derive the calculations in Appendix I from Hsieh et al. (1998) A simple method of sample size calculation for linear and logistic regression (Free PDF by core.ac.uk).

Your result is:

(z_alpha+z_beta)^2 * (1/((p2-p1)^2)) * (p1*(1-p1)+p2*(1-p2)) # 384.5951

matching exactly the normal approximation for a balanced design in Hsieh et al. (Eq. 7 in Appendix I):

beta_star = (p2-p1) / sqrt( (p1* (1-p1) + p2 *(1-p2) )/2)
# Eq. 7
(z_alpha+z_beta)^2 / (p1 * (1-p1) * beta_star^2 ) # 769.1902 

or per arm:

(z_alpha+z_beta)^2 / (p1 * (1-p1) * beta_star^2 )*0.5 # 384.5951 # Hooray!!

But indeed this is still not 390... The reason is that the calculation in Hsieh et al. Eq. 7 (and the ones in the post) are based on normal approximation. When the covariate is a binary variable we can compare the event rates (i.e. binomial proportions) directly. (e.g. See Rosner Fundamentals of Biostatistics (Eq. 10.13 in the 8th edition))

# Eq 9 in the Hsieh et al. (equivalent with FoB 8E, Eq. 10.13)
k = (1-p1)/p1
((1+k) * (z_alpha * sqrt(p1*(1-p1)*(k+1)/1) + # Note we use p1, not p
          z_beta * sqrt(p1*(1-p1) + p2 *(1-p2)/k))^2 / (p1-p2)^2) /2 # 390.0778

As seen, the observed difference is subtle. Finally, to reference how this last result is derived I would suggest to look in Whittemoore (1980) Sample Size for Logistic Regression with Small Response Probability, it is based on asymptotic covariance matrix of the maximum likelihood estimates from the binomial distribution and; it complements the Hsieh et al. paper perfectly.

$\endgroup$
4
  • $\begingroup$ This is the answer I have been looking for. Thank you very much! In general, I have noticed that hypothesis testing is becoming more and more popular as more companies have more data and want to perform A/B tests, but very few of them care about the mathematics under the hood. By the way, you mentioned the book Fundamentals of Biostatistics. Can you recommend it as a reference? I have read mixed reviews on Amazon, but it's often quoted in articles and blog posts. I skimmed through a few pages and it seems that it's just a collection of theorems and results, without proofs $\endgroup$ – Don Draper Apr 19 at 6:41
  • 1
    $\begingroup$ No... It depends a lot on what is your work. If you are starting on A/B tests, probably Kohavi et al. Trustworthy Online Controlled Experiments is a great bet, Kohavi is a Sith lord when it comes to A/B tests. Personally, I have come across a few Biostats resources that I mix-and-match them for my responses. The only Biostats book I have read almost cover-to-cover is Vittinghoff et al. Regression methods in Biostatistics which tells you a lot about the level of rigour I aim for from my main resource. $\endgroup$ – usεr11852 Apr 19 at 9:33
  • $\begingroup$ I have a copy of Kohavi's book, but my main concern is that it totally lacks all the derivations and details I would like to see. It looks more like a handy collection of formulas and guidelines so that you don't have to think about the technical aspect. Honestly, this approach makes me feel uncomfortable. $\endgroup$ – Don Draper Apr 19 at 10:08
  • 1
    $\begingroup$ Hmm. I don't have a go-to reference for something like that. As you saw, even in my answer I was mostly based on papers rather than than a single book. Maybe do a new question here? Reference requests questions are valid. Do mention there which books you have already tried to people can have a good idea of where you are coming from. $\endgroup$ – usεr11852 Apr 19 at 10:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.