When comparing two independent proportions (p2, p1) in an A/B test, null hypothesis states that the difference is 0, while the alternative hypothesis states it is something other than 0, let's say mu1.

From the pooled sample standard deviation, we can say that the distribution for null hypothesis is:

p2-p1 ~ N(0, pooled sample standard deviation)

For the alternative hypothesis, what is the distribution? Is that p2-p1 ~ N(p2-p1, same pooled standard deviation)?

Technically if you look at the distributions below from the power analysis software G*power, I am looking to find values of the blue dotted line, that shows the non central distribution stemming from alternative hypothesis.

I tried to go over Cohen's book (1988) Chapter 6. But I cannot reach an answer how the null hypothesis mu and std are defined.

enter image description here

  • 2
    $\begingroup$ if you're using a normal approx to the binomial to conduct hypothesis testing, we usually don't estimate the variance, but plug in its theoretical value under the given hypothesis using the mean-variance relationship of the binomial. In your case, it gets a little more involved because the two binomials will have different variances. $\endgroup$ Commented Jul 29, 2022 at 20:07
  • $\begingroup$ Upon further studying Cohen's stats book, these are calculated via effect size, that is the arcsin transformation of these proportions for the distribution of the blue line. $\endgroup$
    – kukushkin
    Commented Aug 1, 2022 at 22:17


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.