1
$\begingroup$

I want to calculate the statistical power of a 2 sided A/B test where I have 2 groups of coin tosses (that is, from 2 different coins) and their respective successes. I want to test whether the two coins have different probabilities.

I'm following page 6 of https://ncss-wpengine.netdna-ssl.com/wp-content/themes/ncss/pdf/Procedures/PASS/Tests_for_Two_Proportions.pdf, which follows the paper Chow et al. (2008) (I haven't found it online), and states that:

$$1 - \beta = Pr(Z < \frac{z_\alpha \sigma_{D,p} + (p_1 - p_2)}{\sigma_{D,u}})$$

where:

$n_1$ and $n_2$ are the respective number of coin tosses.

$s_1$ and $s_2$ are the respective successes.

$p_1 = s_1/n_1$, $p_2 = s_2/n_2$.

$q_1 = 1 - p_1, q_2 = 1 - p_2$.

$\sigma_{D,u}$ is the standard unpooled error: $\sqrt{\frac{p_1 q_1}{n_1} + \frac{p_2 q_2}{n_2}}$

$\bar{p} = \frac{n_1 p_1 + n_2 p_2}{n_1 + n_2}$, $\bar{q} = 1 - \bar{p}$.

$\sigma_{D,p}$ is the standard pooled error: $\sqrt{\bar{p} \bar{q} (\frac{1}{n_1} + \frac{1}{n_2})}$

and $z_\alpha$ seems to be the critical value.

I have no problems with the standard errors and I know I'm calculating them correctly. However, I know I'm not getting a correct statistical power as I compared my results to this tool and I'm getting other values.

I suspect I'm calculating $z_\alpha$ incorrectly. The book states the following:

"Find the critical value (or values in the case of a two-sided test) using the standard normal distribution. The critical value is that value of z that leaves exactly the target value of alpha in the tail."

I'm using $\alpha = 0.05$ (I want a confidence level of $0.95$). What value for $z_\alpha$ should I be using and why? I don't understand how to calculate the $z_\alpha$ that this book describes. I don't understand if, it being a 2 sided test, I should be doing 2 calculations.

For the record, let's assume:

$$n_1 = 81000$$

$$n_2 = 80000$$

$$s_1 = 1600$$

$$s_2 = 1696$$

where $n_1$ and $n_2$ are the respective number of coin tosses with each coin, and $s_1$ and $s_2$ are the respective number of successes.

In this example, as far as I know, the correct power should be (according to an online tool), 83.13%.

This it not for homework; I'm implementing a specialized open source tool and want to share it online.

$\endgroup$
2
  • $\begingroup$ Please define symbols that you are using and show formulas for $\sigma_{D,p}, \sigma_{D,u}.$ Questions here should be as self-contained as possible, not relying on users here to find and read relevant parts of a linked (possibly unreviewed) paper. $\endgroup$
    – BruceET
    Nov 17 '20 at 23:01
  • $\begingroup$ you are right, done $\endgroup$ Nov 18 '20 at 0:22
1
$\begingroup$

Your notation, and thus details of your Question, are unclear. Are $S_1$ and $S_2$ observed numbers of successes? If so, I assume you have $n \approx 80\,000$ and want to to know power of a test of $H_0: p = 0.20$ vs. $H_1: p\ne 0.20,$ at the 5% level of significance, to detect differences of about $\pm 0.02.$

Result from Minitab statistical software. If all of that is true, then the following output from a 'power and sample size' procedure in a recent release of Minitab may be relevant. This procedure assumes equal sample sizes in the two groups.

Power and Sample Size 

Test for Two Proportions

Testing comparison p = baseline p (versus ≠)
Calculating power for baseline p = 0.02
α = 0.05

              Sample
Comparison p    Size     Power
       0.022   80000  0.796665

The sample size is for each group.

enter image description here

Note: There are many versions of tests for differences in binomial success probabilities in various statistical software packages, depending on whether a pooled or unpooled standard error is used, whether a continuity correction is used for a normal approximation, and other considerations. Especially using procedures in online sites, you need to know the details of the test for which the power computation is made.

Simulation in R. In R, I used a two-sided prop.test without a continuity correction (on account of large sample sizes) to distinguish between numbers of successes from $\mathsf{Binom}(81000, 0.019)$ and $\mathsf{Binom}(80000, 0.021)$ at the 5% level. In 100,000 such tests there were $81,769$ rejections. So the power of the test is about 81.8%.

set.seed(1117)
pv = replicate(10^5, prop.test( c(rbinom(1,81000,.019),
               rbinom(1,80000,.021)) , c(81000,80000), cor=F)$p.val)
mean(pv <= 0.05)
[1] 0.81769

The first of these 100,000 tests led to rejection of $H_0$ with P-value $0.014 < 0.05 = 5\%,$ as follows:

set.seed(1117)
x1 = rbinom(1, 81000, 0.019)
x2 = rbinom(1, 80000, 0.021)
x1; x2
[1] 1500
[1] 1663


prop.test(c(x1,x2), c(81000,80000), cor=F)

        2-sample test for equality of proportions 
        without continuity correction

data:  c(x1, x2) out of c(81000, 80000)
X-squared = 10.759, df = 1, p-value = 0.001038
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.0036252315 -0.0009127315
sample estimates:
    prop 1     prop 2 
0.01851852 0.02078750 
$\endgroup$
5
  • $\begingroup$ Hi, I really appreciate your answer, but I'm not looking for the statistical power of this particular example. My question (as I stated) is the meaning of $z_\alpha$ at page 6 of the link and how to calculate it in the pooled version of a 2-sided A/B test. $\endgroup$ Nov 17 '20 at 23:34
  • $\begingroup$ I edited my question in an effort to make it clearer $\endgroup$ Nov 17 '20 at 23:35
  • $\begingroup$ Moreover, please notice that in my scenario I have an unequal sample size $\endgroup$ Nov 17 '20 at 23:36
  • $\begingroup$ Yes. You may have trouble finding software or an online calculator that accepts different $n$'s as input. The most powerful configuration is a balanced design. If the two $n$/s are nearly the same, then use the smaller for such procedures. If you can find a formula that works for the style of test you're using, then you can use different sample sizes. And (as above) you can use different sample sizes in a simulation. $\endgroup$
    – BruceET
    Nov 17 '20 at 23:53
  • $\begingroup$ This tool: abtestguide.com/calc accepts both n's, I think - and for what I am doing I would like to do the same $\endgroup$ Nov 18 '20 at 0:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.