AB-Testing on Website: Binomial Test vs Chi2 or Z-Test I have data from a AB-Test on a website:





Version A
Version B




#Visitors
10010
9912


#Conversion
320
275




I want to test the null hypothesis that the difference in conversion is due to chance.
While doing research, I noticed most often a Z-Test or chi2 test is used for these website AB-Test.
Why would you use for such a problem a Z-Test or chi2-test? Wouldn't a binomial test the better solution (since Z-Test or chi2 are only approximation based on the central limit theorem)?
 A: Here is a output from Minitab for a test of binomial proportions.
(I used the version in which the two samples are taken separately
to estimate the standard error.)
Test and CI for Two Proportions 

Sample    X      N  Sample p
1       320  10010  0.031968
2       275   9912  0.027744

Difference = p (1) - p (2)
Estimate for difference:  0.00422388
95% CI for difference:  (-0.000501583, 0.00894935)
Test for difference = 0 (vs ≠ 0):  Z = 1.75  P-Value = 0.080

Fisher’s exact test: P-Value = 0.081

Notes:
(1) The two-sided 95% confidence interval
contains $0.$
(2) Minitab output also shows the P-value
for Fisher's Exact Test, which is based on a hypergeometric distribution.
(3) The version of the test with a pooled estimate of the
standard error also gives P-value 0.080.
(4) All of the above is for two-sided tests. If you intended to test
against the alternative that A has a higher conversion rate, then the
P-value is 0.040 and the confidence interval is one-sided, giving a (very small) lower
bound for the difference.
95% lower bound for difference:  0.000258147

(5) All tests above (except Fisher's Exact Test) use normal approximations, which should be quite accurate for sample sizes as large as yours.
(6) A chi-squared test in R (inherently two-sided) gives
about the same P-value as the other two-sided tests:
TBL = rbind(c(320, 275), c(10010-320, 9912-275))
TBL
     [,1] [,2]
[1,]  320  275
[2,] 9690 9637

chisq.test(TBL, cor=F) # without Yates correction: large samples

        Pearson's Chi-squared test

data:  TBL
X-squared = 3.0667, df = 1, p-value = 0.07991

