# How to calculate z-score of binomial values for an ab test

I ran an ab-test and now I'm trying to calculate the pvalue. The test and control are binomial and these are the values To get the variance, I used the variance formula from here https://stattrek.com/probability-distributions/binomial.aspx This gives me the values in orange: I should be able to get the z-value by just doing a z-score calculation as show in this link:(https://courses.lumenlearning.com/introstats1/chapter/two-population-means-with-known-standard-deviations/) shows that I can calculate the z-score by For me, this should be(0.12-0.09)/SQRT(43.8/550+49/450)=.069093. If I look this up (https://www.socscistatistics.com/pvalues/normaldistribution.aspx) this gives a p-value of .472.

I don't think this is right because if I try a proportion test in Python, I get a different value

from statsmodels.stats.proportion import proportions_ztest
stat, pval = proportions_ztest(count=np.array([48,56]),
nobs=np.array([550, 450]),
alternative='two-sided',
prop_var=False
)
print (stat, pval)


I get a t-stat of 1.916 and a p-value of 0.055.

I'm not sure where I have a made a mistake in calculating, can anyone help?

• Not sure you really made a mistake. There are several versions of this test and you may have gotten confused using two versions at once. See my Answer. Jun 23, 2021 at 0:22
• Thank you so much! This is great! Jun 23, 2021 at 3:20

Several methods are in common use for testing $$H_0: p_1=p_2$$ against $$H_a: p_l\ne p_2.$$

One method (called 'pooled') uses the null hypothesis $$p_1=p_2=p$$ to estimate $$\hat p = \frac{x_1+x_2}{n_1+n_2} = \frac{48+56}{550+450} = \frac{104}{1000} = 0.104$$ and then base the (estimated) standard error of $$\hat p_1 - \hat p_2$$ (denominator of $$Z)$$ as $$\sqrt{\hat p(1-\hat p)(1/n_1+1/n_2)}.$$ See NIST handbook.

Another method is to estimate $$Var(\hat p_i), i = 1,2$$ separately as $$\hat\sigma_i =\hat p_i(1-\hat p_i)/n_i,$$ respectively. Then the (estimated) standard error of $$\hat p_1 - \hat p_2$$ is $$\sqrt{\hat\sigma_1 + \hat\sigma_2}.$$ [See, e.g., Ott & Longnecker: Intro. to Stat. Meth. and Data Analysis, Ch 10.] An advantage of this method is that the estimated standard error is also appropriate for making a CI of $$p_1 - p_2.$$

Both methods have been used in software implementations of the test for two binomial proportions, sometimes with various continuity corrections. Minitab software uses this method as default, but allows the user to specify the pooled method.

From a recent release of Minitab, here is output from the 'separate variances' method, with P-value $$0.059.$$

Test and CI for Two Proportions

Sample   X    N  Sample p
1       48  550  0.087273
2       56  450  0.124444

Difference = p (1) - p (2)
Estimate for difference:  -0.0371717
95% CI for difference:  (-0.0757267, 0.00138324)
Test for difference = 0 (vs ≠ 0):
Z = -1.89  P-Value = 0.059


Also, using the 'pooled' method, the P-value is $$0.055.$$ However, the 95% CI for $$p_1 - p_2$$ uses the separate variances method, so the CI is exactly the same in both outputs.

Test and CI for Two Proportions

Sample   X    N  Sample p
1       48  550  0.087273
2       56  450  0.124444

Difference = p (1) - p (2)
Estimate for difference:  -0.0371717
95% CI for difference:  (-0.0757267, 0.00138324)
Test for difference = 0 (vs ≠ 0):
Z = -1.92  P-Value = 0.055


Output from the procedure prop.test in R, agrees with the separate variances version of Minitab except for differences in rounding.

prop.test(c(48,56), c(550,450), cor=F)

2-sample test for equality of proportions
without continuity correction

data:  c(48, 56) out of c(550, 450)
X-squared = 3.6699, df = 1, p-value = 0.0554
alternative hypothesis: two.sided
95 percent confidence interval:
-0.07572667  0.00138324
sample estimates:
prop 1     prop 2
0.08727273 0.12444444


Essentially the same test can be done in R, using a chi-squared test (chisq.test), which uses for input a $$2\times 2$$ table of successes and failures.

TBL = rbind(c(48,56), c(550-48, 450-56)); TBL
[,1] [,2]
[1,]   48   56
[2,]  502  394

chisq.test(TBL, cor=F)

Pearson's Chi-squared test

data:  TBL
X-squared = 3.6699, df = 1, p-value = 0.0554


On account of the relatively small counts, some statisticians would use Fisher's exact test, which gives P-value $$0.061,$$ also failing to reject at the 5% level. [See Wikipedia.] This P-value for Fisher's exact test is also shown in Minitab printouts of the pooled and separate variances methods, but was omitted above.

fisher.test(TBL)

Fisher's Exact Test for Count Data

data:  TBL
p-value = 0.06102
alternative hypothesis:
true odds ratio is not equal to 1
95 percent confidence interval:
0.4375136 1.0319594
sample estimates:
odds ratio
0.6730183

• Thank you! This was so helpful Jun 29, 2021 at 16:35