I'm using the statistical software G*Power to find the minimum sample size needed to detect whether or not the difference in two proportions is statistically significant in an AB test I'm setting up. I have found plenty of sources stating that the chi-squared and the z-test are equivalent in this circumstance (i.e. Example)
I am, however, getting different sample size requirements depending on which test I use (chisq x z-test). In the image below, the colors refer to whether we are under $H_0$ (gray) or under $H_1$ (blue). The upper part shows counts and the lower part shows percentages.
I toyed around with a baseline conversion of 40% and that set the tone for the counts you see in gray, under $H_0$; meanwhile, I decided to experiment with observed counts of 41 in the control group and 45 in the treatment group, setting that as a small-effect scenario I would like to be able to detect in my test.
So, with that scenario in mind, I used G*Power to return the minimum sample sizes I would need if using a Z-Test or a Chi-squared test, and was hoping they both would tell me the same thing, roughly. These were the results:
- While the Chisq test tells me a sample of 2,400 observations are enough:
- The z-test test tells me a sample of 7958 is required.
If I'm using the same parameters (confidence level, power, two-tail test for the z-test), aren't these tests supposed to require the same sample and produce roughly the same critical values?
Is this discrepancy a result from my poor use of the software or did I misunderstand which circumstances make these tests equivalent?