Got stuck on an issue. We are doing a simple A/B, where treatment B has some updated legal language. We want to have appropriate statistical confidence that treatment B will not lower performance. How do I structure such a test? In previous tests I have been running simulations to check how often we'd be able to detect a difference if it exists. Now I want to know the required sample size to be able to say "treament B is not making it perform worse", but I'm unsure of how to approach it...

Some numbers to make it easier to discuss. Treatment A has 14% retention rate. We want to be able to say with statistical confidence that treatment B does not reduce that (i.e. has at least as high retention rate).

How do I calculate the needed sample size from this? How do I evaluate results?


Let $p_A$ and $p_B$ be the proportion of subjects exhibiting the outcome of interest in groups A and B, respectively. Further, let the sample sizes of groups A and B be $n_A$ and $n_B$, respectively.

We specify the following:

  • $R=n_A/n_B$ is the ratio of sample sizes;
  • $\bar{p}=(p_A n_A + p_B n_B)/(n_A + n_B)$;
  • $\alpha$ is the level of significance and $z_{(1-\alpha)}$ is the $(1-\alpha)^{th}$ quantile of the standard normal distribution; and
  • $\beta$ is the type II error rate and $z_\beta$ is the $\beta^{th}$ quantile of the standard normal distribution;

The sample size for a one-sided test of proportion is

$n=\dfrac{\left[ z_{(1-\alpha)} \sqrt{\bar{p}(1-\bar{q})} - z_\beta \sqrt{p_A(1-p_A)/(1+R) + p_B(1-p_B)R/(1+R)} \ \right]^2}{R(p_B-p_A)^2/(1+R)^2}$

If $R=1$, then $n_A=n_B=n/2$.

Without knowledge of any special circumstances, you would assess this using the chi-squared test.

Good luck!


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.