2
$\begingroup$

I'm using the statistical software G*Power to find the minimum sample size needed to detect whether or not the difference in two proportions is statistically significant in an AB test I'm setting up. I have found plenty of sources stating that the chi-squared and the z-test are equivalent in this circumstance (i.e. Example)

I am, however, getting different sample size requirements depending on which test I use (chisq x z-test). In the image below, the colors refer to whether we are under $H_0$ (gray) or under $H_1$ (blue). The upper part shows counts and the lower part shows percentages.

enter image description here

I toyed around with a baseline conversion of 40% and that set the tone for the counts you see in gray, under $H_0$; meanwhile, I decided to experiment with observed counts of 41 in the control group and 45 in the treatment group, setting that as a small-effect scenario I would like to be able to detect in my test.

So, with that scenario in mind, I used G*Power to return the minimum sample sizes I would need if using a Z-Test or a Chi-squared test, and was hoping they both would tell me the same thing, roughly. These were the results:

  • While the Chisq test tells me a sample of 2,400 observations are enough:

enter image description here

  • The z-test test tells me a sample of 7958 is required.

enter image description here

If I'm using the same parameters (confidence level, power, two-tail test for the z-test), aren't these tests supposed to require the same sample and produce roughly the same critical values?

Is this discrepancy a result from my poor use of the software or did I misunderstand which circumstances make these tests equivalent?

$\endgroup$
0

2 Answers 2

1
$\begingroup$

There are different chi-square tests: Chi-square test: difference between goodness-of-fit test and test of independence

Let's focus on the 2×2 case since you want to compare the conversion probabilities of two groups (treatment and control) in an A/B test.

  • The test for homogeneity of proportions tests whether the control and treatment group have the same probability of conversion. This is the test for difference in two proportions that you intend to perform.
  • The test for goodness of fit tests whether the control and treatment group have the specified probabilities of conversion.

Intuitively, you need a smaller sample size to detect any kind of deviation from the specified proportions $(p_1,1-p_1,p_2,1-p_2)$ than to estimate the two proportions and compare them.

We can confirm by computing the sample size required for the two-sample binomial test with the Hmisc package.

alpha <- 0.05
power <- 0.95

p1 <- 0.41
p2 <- 0.45

Hmisc::bsamsize(p1, p2, alpha = alpha, power = power)
#>       n1       n2 
#> 3978.289 3978.289
$\endgroup$
1
  • $\begingroup$ this explains a lot, thank you. $\endgroup$ Commented Aug 22, 2022 at 12:30
1
$\begingroup$

Probably the reason being that Z test is based on a Normal approximation of Binomial distributions. The Chi square test is more based on the actual data and so should be the correct one.

$\endgroup$
2
  • 1
    $\begingroup$ 1. That normal approximation to the binomial will be excellent when n's are in the thousands unless the probability parameters were extremely large or small. If that was the cause, it would not make this level of difference. 2. The chi-squared test is not "more based on actual data". The distribution of a chi-square statistic here also relies on a continuous, asymptotic approximation to a discrete distribution of a test statistic. The explanation will lay elsewhere (e.g. that the two tests are not actually testing quite the same thing in this case) $\endgroup$
    – Glen_b
    Commented Aug 21, 2022 at 22:55
  • $\begingroup$ ... or perhaps that some input doesn't actually correspond, etc. $\endgroup$
    – Glen_b
    Commented Aug 21, 2022 at 23:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.