A/B test sample size formula confusion I was trying to understand the math behind some commonly used calculators and formulas for A/B tests and it seems that there might be some variations. Ideally, I would like to understand how each of them is derived and under what assumptions.
$$\left(\frac{\Phi_{1-\alpha/2}^{-1}\sqrt{2\bar p(1-\bar p)}+\Phi_{1 -\beta}^{-1}\sqrt{p_1(1-p_1)+p_2(1-p_2)}}{|p_2-p_1|}\right)^2$$ where $$ \bar p = \frac{p_1 + p_2}{2}$$ as far as I understand.
This is the formula I found here, for example:
https://jeffshow.com/caculate-abtest-required-sample-size.html
It's in Chinese, but from what I read after using Google Translate, this is exactly that type of calculation.
Honestly speaking, I have no idea where this formula comes from.
In the general case, the formula for the sample size is:
$$n \ge 2 \Big(\frac{\Phi^{-1}(1-\alpha/2)+\Phi^{-1}(1-\beta)}{\Delta/\sigma}\Big)^2$$
Under the assumption of equal variances, we can calculate the pooled variance as:
$$\sigma_p=\sqrt{\dfrac{(n_1-1)s_1^2+(n_2-1)s^2_2}{n_1+n_2-2}}$$, which can be simplified given that $n_1 = n_2$ as $\sigma_p = \sqrt{\frac{p_1\cdot(1-p_1) + p_2 \cdot (1-p_2)}{2}}$.
Plugging that into the formula for the general case, we get:
$$n \geq \frac{p_1\cdot(1-p_1) + p_2 \cdot (1-p_2)}{(p_2-p_1)^2} \cdot (z_{\frac{\alpha}{2}} + z_{\beta})^2$$
The results, however, are not the same when using these two formulas. As far as I understand, the famous Evan Miller calculator is based on the first formula.
For $p_1=0.5$ and $p_2=0.6$, my formula gives $n=384.16$ whereas this calculator reports $390$ as the answer.
Can you please clarify it a bit? I believe there might be some assumptions or, even worse, something really trivial I fail to take into consideration.
Thanks!
 A: The Maths is correct; the interpretation of what is the baseline needs a bit adjusting to get back to the EM result.
When comparing two arbitrary proportion coming from equal sized population, indeed the base-line proportion is $\bar{p}$ is $\frac{p_1 + p_2}{2}$. When though making an A/B test the base-line proportion is $p_1$. As such if we use $\bar{p}=p_1$ then we can get the results in question:
p1 = 0.5
p2 = 0.6
delta = abs(p1-p2)
alpha_level = 0.05
beta_level = 0.80
z_alpha = qnorm(1.0-alpha_level/2)
z_beta = qnorm(beta_level)

sd1_ab = sqrt(2*p1*(1.0-p1))
sd2 = sqrt(p1*(1-p1)+p2*(1-p2))
((z_alpha * sd1_ab + z_beta * sd2) / abs(delta))^2 # 390.0778 as expected

Now, what you have done though is to correctly derive the calculations in Appendix I from Hsieh et al. (1998) A simple method of sample size calculation for linear and logistic regression (Free PDF by core.ac.uk).
Your result is:
(z_alpha+z_beta)^2 * (1/((p2-p1)^2)) * (p1*(1-p1)+p2*(1-p2)) # 384.5951

matching exactly the normal approximation for a balanced design in Hsieh et al. (Eq. 7 in Appendix I):
beta_star = (p2-p1) / sqrt( (p1* (1-p1) + p2 *(1-p2) )/2)
# Eq. 7
(z_alpha+z_beta)^2 / (p1 * (1-p1) * beta_star^2 ) # 769.1902 

or per arm:
(z_alpha+z_beta)^2 / (p1 * (1-p1) * beta_star^2 )*0.5 # 384.5951 # Hooray!!

But indeed this is still not 390... The reason is that the calculation in Hsieh et al. Eq. 7 (and the ones in the post) are based on normal approximation. When the covariate is a binary variable we can compare the event rates (i.e. binomial proportions) directly. (e.g. See Rosner Fundamentals of Biostatistics (Eq. 10.13 in the 8th edition))
# Eq 9 in the Hsieh et al. (equivalent with FoB 8E, Eq. 10.13)
k = (1-p1)/p1
((1+k) * (z_alpha * sqrt(p1*(1-p1)*(k+1)/1) + # Note we use p1, not p
          z_beta * sqrt(p1*(1-p1) + p2 *(1-p2)/k))^2 / (p1-p2)^2) /2 # 390.0778

As seen, the observed difference is subtle. Finally, to reference how this last result is derived I would suggest to look in Whittemoore (1980) Sample Size for Logistic Regression with Small Response Probability, it is based on asymptotic covariance matrix of the maximum likelihood estimates from the binomial distribution and; it complements the Hsieh et al. paper perfectly.
