# Significance and power for binomial A/B test

Assume A/B test with control sample $X = \left(X_1, \dots, X_n\right)$ and treatment $Y = \left( Y_1, \dots, Y_m \right)$.

Let $X_i$ be i.i.d. $\text{Bernoulli}(\mu_x)$ and $Y_j$ be i.i.d. $\text{Bernoulli}(\mu_y)$. Both parameters are unknown, the two samples are indepenent (unpaired).

I would like to compare true means of these populations. The very goal is to investigate the hypothesis $H_0: \, \mu_y \geq \mu_x$ against the alternative $H_1: \, \mu_y \lt \mu_x$ but I have found no methods to test it. Any links are appreciated.

Since then let's assume $H_0: \, \mu_y = \mu_x$ against $H_1: \, \mu_y \gt \mu_x$. I would like to get a reasonable exact or asymptotical test with the known distribution of the test statistic under the hypothesis and the alternative as I need to calculate the siginificance and power of such a test. Unfortunately almost all the popular internet guides are rather informal, have lacks in proofs and don't deal with power calculating.

\begin{align*} H_0&\colon \mu_y-\mu_x\leq 0, \\ H_1&\colon \mu_y-\mu_x > 0, \\ Z &= \frac{\hat{\mu}_y-\hat{\mu}_x} {\sqrt{ \frac{\hat{\mu}_y\left(1-\hat{\mu}_y\right)}{m} + \frac{\hat{\mu}_x\left(1-\hat{\mu}_x\right)}{n}} } \approx\sim N\left(0,1\right)\text{ under }H_0,\\ power&=\Phi\left( \frac{\mu_y-\mu_x} {\sqrt{ \frac{\mu_y\left(1-\mu_y\right)}{m} + \frac{\mu_x\left(1-\mu_x\right)}{n}} } - z_\alpha\right)\end{align*} (note that the last formula involves actual parameter values, not their sample estimates).
• thanks for your answer. Unfortunately I don't have access to this book. It's not clear for me why $Z$ is normally distributed under $H_0$. I see the normal variable in the numerator and a sum of $\chi^2$ and normally distributed under the root if one applies normal appproximation of binomial variables. Commented Jun 2, 2017 at 13:51
• I have no idea why in this article $Z \approx N(0, 1)$. The denominator - SE - is a r.v. Why is this ratio close to a normal variable? Commented Jun 2, 2017 at 14:07