Calculate power of 2 sided Z A/B test - pooled version

Question

I want to calculate the statistical power of a 2 sided A/B test where I have 2 groups of coin tosses (that is, from 2 different coins) and their respective successes. I want to test whether the two coins have different probabilities.

I'm following page 6 of https://ncss-wpengine.netdna-ssl.com/wp-content/themes/ncss/pdf/Procedures/PASS/Tests_for_Two_Proportions.pdf, which follows the paper Chow et al. (2008) (I haven't found it online), and states that:

$$1 - \beta = Pr(Z < \frac{z_\alpha \sigma_{D,p} + (p_1 - p_2)}{\sigma_{D,u}})$$

where:

$n_1$ and $n_2$ are the respective number of coin tosses.

$s_1$ and $s_2$ are the respective successes.

$p_1 = s_1/n_1$, $p_2 = s_2/n_2$.

$q_1 = 1 - p_1, q_2 = 1 - p_2$.

$\sigma_{D,u}$ is the standard unpooled error: $\sqrt{\frac{p_1 q_1}{n_1} + \frac{p_2 q_2}{n_2}}$

$\bar{p} = \frac{n_1 p_1 + n_2 p_2}{n_1 + n_2}$, $\bar{q} = 1 - \bar{p}$.

$\sigma_{D,p}$ is the standard pooled error: $\sqrt{\bar{p} \bar{q} (\frac{1}{n_1} + \frac{1}{n_2})}$

and $z_\alpha$ seems to be the critical value.

I have no problems with the standard errors and I know I'm calculating them correctly. However, I know I'm not getting a correct statistical power as I compared my results to this tool and I'm getting other values.

I suspect I'm calculating $z_\alpha$ incorrectly. The book states the following:

"Find the critical value (or values in the case of a two-sided test) using the standard normal distribution. The critical value is that value of z that leaves exactly the target value of alpha in the tail."

I'm using $\alpha = 0.05$ (I want a confidence level of $0.95$). What value for $z_\alpha$ should I be using and why? I don't understand how to calculate the $z_\alpha$ that this book describes. I don't understand if, it being a 2 sided test, I should be doing 2 calculations.

For the record, let's assume:

$$n_1 = 81000$$

$$n_2 = 80000$$

$$s_1 = 1600$$

$$s_2 = 1696$$

where $n_1$ and $n_2$ are the respective number of coin tosses with each coin, and $s_1$ and $s_2$ are the respective number of successes.

In this example, as far as I know, the correct power should be (according to an online tool), 83.13%.

This it not for homework; I'm implementing a specialized open source tool and want to share it online.

Answer 1

Your notation, and thus details of your Question, are unclear. Are $S_1$ and $S_2$ observed numbers of successes? If so, I assume you have $n \approx 80\,000$ and want to to know power of a test of $H_0: p = 0.20$ vs. $H_1: p\ne 0.20,$ at the 5% level of significance, to detect differences of about $\pm 0.02.$

Result from Minitab statistical software. If all of that is true, then the following output from a 'power and sample size' procedure in a recent release of Minitab may be relevant. This procedure assumes equal sample sizes in the two groups.

Power and Sample Size 

Test for Two Proportions

Testing comparison p = baseline p (versus ≠)
Calculating power for baseline p = 0.02
α = 0.05

              Sample
Comparison p    Size     Power
       0.022   80000  0.796665

The sample size is for each group.

Note: There are many versions of tests for differences in binomial success probabilities in various statistical software packages, depending on whether a pooled or unpooled standard error is used, whether a continuity correction is used for a normal approximation, and other considerations. Especially using procedures in online sites, you need to know the details of the test for which the power computation is made.

Simulation in R. In R, I used a two-sided prop.test without a continuity correction (on account of large sample sizes) to distinguish between numbers of successes from $\mathsf{Binom}(81000, 0.019)$ and $\mathsf{Binom}(80000, 0.021)$ at the 5% level. In 100,000 such tests there were $81,769$ rejections. So the power of the test is about 81.8%.

set.seed(1117)
pv = replicate(10^5, prop.test( c(rbinom(1,81000,.019),
               rbinom(1,80000,.021)) , c(81000,80000), cor=F)$p.val)
mean(pv <= 0.05)
[1] 0.81769

The first of these 100,000 tests led to rejection of $H_0$ with P-value $0.014 < 0.05 = 5\%,$ as follows:

set.seed(1117)
x1 = rbinom(1, 81000, 0.019)
x2 = rbinom(1, 80000, 0.021)
x1; x2
[1] 1500
[1] 1663


prop.test(c(x1,x2), c(81000,80000), cor=F)

        2-sample test for equality of proportions 
        without continuity correction

data:  c(x1, x2) out of c(81000, 80000)
X-squared = 10.759, df = 1, p-value = 0.001038
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.0036252315 -0.0009127315
sample estimates:
    prop 1     prop 2 
0.01851852 0.02078750