# Proof of the standard error of the distribution between two normal distributions in A/B testing

Classic A/B test suppose that there are two independent experiments, each with $$n_1$$ and $$n_2$$ observations, for which we are interested in an event following a binomial distribution $$\mathcal{B}(n_1,p_1)$$ and $$\mathcal{B}(n_2,p_2)$$. If we suppose that the central limit theorem is valid for each experiment then, by respectively dividing by $$n_1$$ and $$n_2$$, we can assume that the event in each experiment will follow a normal distribution $$\mathcal{N}(\hat{p_1}, \frac{\hat{p_1}(1-\hat{p_1})}{n_1})$$ and $$\mathcal{N}(\hat{p_2}, \frac{\hat{p_2}(1-\hat{p_2})}{n_2})$$.

The A/B test looks for a significant difference between $$p_1$$ and $$p_2$$ by taking the difference between the two former distributions. This gives a normal distribution $$\mathcal{N}(\hat{p_1}-\hat{p_2}, \hat{p}(1-\hat{p})(\frac{1}{n_1}+\frac{1}{n_2}))$$, where $$\hat{p}=\frac{n_1\hat{p_1}+n_2\hat{p_2}}{n_1+n_2}$$.

The formula of this variance is given in various ressources such as :

I know that the variance given by the difference of two normal distributions is equal to the sum of the variances, and I would like to prove the formula of the variance: $$\hat{p}(1-\hat{p})(\frac{1}{n_1}+\frac{1}{n_2})=\frac{\hat{p_1}(1-\hat{p_1})}{n_1} + \frac{\hat{p_2}(1-\hat{p_2})}{n_2}$$

• If you consider some extreme examples where the $n_i$ are either $1$ or very large, it should become apparent that what you hope to prove just isn't true.
– whuber
Aug 6, 2018 at 22:10

As whuber pointed out the stated equality is false in general. What the expression $\hat{p} (1 - \hat{p})(n^{-1}_{1} + n^{-1}_{2})$ represents is our estimate of the variance of the test statistic when both probabilities are equal, since that is the null hypothesis of the test.
If we assume that $p_1 = p_2$ then our best guess of this common probability is the pooled sample proportion $\hat{p}$, and using this to estimate the variance of $\hat{p}_1 - \hat{p}_2$ gives the formula you reference after assuming independence.