Assume A/B test with control sample $X = \left(X_1, \dots, X_n\right)$ and treatment $Y = \left( Y_1, \dots, Y_m \right)$.

Let $X_i$ be i.i.d. $\text{Bernoulli}(\mu_x)$ and $Y_j$ be i.i.d. $\text{Bernoulli}(\mu_y)$. Both parameters are unknown, the two samples are indepenent (unpaired).

I would like to compare true means of these populations. The very goal is to investigate the hypothesis $H_0: \, \mu_y \geq \mu_x$ against the alternative $H_1: \, \mu_y \lt \mu_x$ but I have found no methods to test it. Any links are appreciated.

Since then let's assume $H_0: \, \mu_y = \mu_x$ against $H_1: \, \mu_y \gt \mu_x$. I would like to get a reasonable exact or asymptotical test with the known distribution of the test statistic under the hypothesis and the alternative as I need to calculate the siginificance and power of such a test. Unfortunately almost all the popular internet guides are rather informal, have lacks in proofs and don't deal with power calculating.

Thanks for your help.


1 Answer 1


There are nice analytical formulas just for your case in Chow, Sample Size Calculations in Clinical Research (2nd ed., 2008), 4.2.2:

$$\begin{align*} H_0&\colon \mu_y-\mu_x\leq 0, \\ H_1&\colon \mu_y-\mu_x > 0, \\ Z &= \frac{\hat{\mu}_y-\hat{\mu}_x} {\sqrt{ \frac{\hat{\mu}_y\left(1-\hat{\mu}_y\right)}{m} + \frac{\hat{\mu}_x\left(1-\hat{\mu}_x\right)}{n}} } \approx\sim N\left(0,1\right)\text{ under }H_0,\\ power&=\Phi\left( \frac{\mu_y-\mu_x} {\sqrt{ \frac{\mu_y\left(1-\mu_y\right)}{m} + \frac{\mu_x\left(1-\mu_x\right)}{n}} } - z_\alpha\right)\end{align*}$$ (note that the last formula involves actual parameter values, not their sample estimates).

  • $\begingroup$ thanks for your answer. Unfortunately I don't have access to this book. It's not clear for me why $Z$ is normally distributed under $H_0$. I see the normal variable in the numerator and a sum of $\chi^2$ and normally distributed under the root if one applies normal appproximation of binomial variables. $\endgroup$ Commented Jun 2, 2017 at 13:51
  • $\begingroup$ It comes from the normal approximation for the binomial distribution, some more details here, for example: 0agr.ru/wiki/index.php/… $\endgroup$ Commented Jun 2, 2017 at 13:57
  • $\begingroup$ I have no idea why in this article $Z \approx N(0, 1)$. The denominator - SE - is a r.v. Why is this ratio close to a normal variable? $\endgroup$ Commented Jun 2, 2017 at 14:07
  • $\begingroup$ hmm I've just found this book - it seems to have no proofs unfortunately... $\endgroup$ Commented Jun 2, 2017 at 14:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.