How to conduct test of proportions and calculate the needed sample size?

Question

Got stuck on an issue. We are doing a simple A/B, where treatment B has some updated legal language. We want to have appropriate statistical confidence that treatment B will not lower performance. How do I structure such a test? In previous tests I have been running simulations to check how often we'd be able to detect a difference if it exists. Now I want to know the required sample size to be able to say "treament B is not making it perform worse", but I'm unsure of how to approach it...

Some numbers to make it easier to discuss. Treatment A has 14% retention rate. We want to be able to say with statistical confidence that treatment B does not reduce that (i.e. has at least as high retention rate).

How do I calculate the needed sample size from this? How do I evaluate results?

Answer 1

Let $p_A$ and $p_B$ be the proportion of subjects exhibiting the outcome of interest in groups A and B, respectively. Further, let the sample sizes of groups A and B be $n_A$ and $n_B$, respectively.

We specify the following:

• $R=n_A/n_B$ is the ratio of sample sizes;
• $\bar{p}=(p_A n_A + p_B n_B)/(n_A + n_B)$;
• $\alpha$ is the level of significance and $z_{(1-\alpha)}$ is the $(1-\alpha)^{th}$ quantile of the standard normal distribution; and
• $\beta$ is the type II error rate and $z_\beta$ is the $\beta^{th}$ quantile of the standard normal distribution;

The sample size for a one-sided test of proportion is

$n=\dfrac{\left[ z_{(1-\alpha)} \sqrt{\bar{p}(1-\bar{q})} - z_\beta \sqrt{p_A(1-p_A)/(1+R) + p_B(1-p_B)R/(1+R)} \ \right]^2}{R(p_B-p_A)^2/(1+R)^2}$

If $R=1$, then $n_A=n_B=n/2$.

Without knowledge of any special circumstances, you would assess this using the chi-squared test.

Good luck!