2
$\begingroup$

I ran an ab-test and now I'm trying to calculate the pvalue. The test and control are binomial and these are the values enter image description here

To get the variance, I used the variance formula from here https://stattrek.com/probability-distributions/binomial.aspx

enter image description here

This gives me the values in orange: enter image description here

I should be able to get the z-value by just doing a z-score calculation as show in this link:(https://courses.lumenlearning.com/introstats1/chapter/two-population-means-with-known-standard-deviations/) shows that I can calculate the z-score by

enter image description here

For me, this should be(0.12-0.09)/SQRT(43.8/550+49/450)=.069093. If I look this up (https://www.socscistatistics.com/pvalues/normaldistribution.aspx) this gives a p-value of .472.

I don't think this is right because if I try a proportion test in Python, I get a different value

from statsmodels.stats.proportion import proportions_ztest
stat, pval = proportions_ztest(count=np.array([48,56]),
                               nobs=np.array([550, 450]),
                               alternative='two-sided',
                               prop_var=False
                              )
print (stat, pval)

I get a t-stat of 1.916 and a p-value of 0.055.

I'm not sure where I have a made a mistake in calculating, can anyone help?

$\endgroup$
2
  • 1
    $\begingroup$ Not sure you really made a mistake. There are several versions of this test and you may have gotten confused using two versions at once. See my Answer. $\endgroup$
    – BruceET
    Jun 23, 2021 at 0:22
  • 1
    $\begingroup$ Thank you so much! This is great! $\endgroup$ Jun 23, 2021 at 3:20

1 Answer 1

1
$\begingroup$

Several methods are in common use for testing $H_0: p_1=p_2$ against $H_a: p_l\ne p_2.$

One method (called 'pooled') uses the null hypothesis $p_1=p_2=p$ to estimate $\hat p = \frac{x_1+x_2}{n_1+n_2} = \frac{48+56}{550+450} = \frac{104}{1000} = 0.104$ and then base the (estimated) standard error of $\hat p_1 - \hat p_2$ (denominator of $Z)$ as $\sqrt{\hat p(1-\hat p)(1/n_1+1/n_2)}.$ See NIST handbook.

Another method is to estimate $Var(\hat p_i), i = 1,2$ separately as $\hat\sigma_i =\hat p_i(1-\hat p_i)/n_i,$ respectively. Then the (estimated) standard error of $\hat p_1 - \hat p_2$ is $\sqrt{\hat\sigma_1 + \hat\sigma_2}.$ [See, e.g., Ott & Longnecker: Intro. to Stat. Meth. and Data Analysis, Ch 10.] An advantage of this method is that the estimated standard error is also appropriate for making a CI of $p_1 - p_2.$

Both methods have been used in software implementations of the test for two binomial proportions, sometimes with various continuity corrections. Minitab software uses this method as default, but allows the user to specify the pooled method.

From a recent release of Minitab, here is output from the 'separate variances' method, with P-value $0.059.$

Test and CI for Two Proportions 

Sample   X    N  Sample p
1       48  550  0.087273
2       56  450  0.124444

Difference = p (1) - p (2)
Estimate for difference:  -0.0371717
95% CI for difference:  (-0.0757267, 0.00138324)
Test for difference = 0 (vs ≠ 0):  
  Z = -1.89  P-Value = 0.059

Also, using the 'pooled' method, the P-value is $0.055.$ However, the 95% CI for $p_1 - p_2$ uses the separate variances method, so the CI is exactly the same in both outputs.

Test and CI for Two Proportions 

Sample   X    N  Sample p
1       48  550  0.087273
2       56  450  0.124444

Difference = p (1) - p (2)
Estimate for difference:  -0.0371717
95% CI for difference:  (-0.0757267, 0.00138324)
Test for difference = 0 (vs ≠ 0):  
  Z = -1.92  P-Value = 0.055

Output from the procedure prop.test in R, agrees with the separate variances version of Minitab except for differences in rounding.

prop.test(c(48,56), c(550,450), cor=F)

    2-sample test for equality of proportions 
    without continuity correction

data:  c(48, 56) out of c(550, 450)
X-squared = 3.6699, df = 1, p-value = 0.0554
alternative hypothesis: two.sided
95 percent confidence interval:
 -0.07572667  0.00138324
 sample estimates:
    prop 1     prop 2 
0.08727273 0.12444444 

Essentially the same test can be done in R, using a chi-squared test (chisq.test), which uses for input a $2\times 2$ table of successes and failures.

TBL = rbind(c(48,56), c(550-48, 450-56)); TBL           
     [,1] [,2]
[1,]   48   56
[2,]  502  394

chisq.test(TBL, cor=F)

        Pearson's Chi-squared test

data:  TBL
X-squared = 3.6699, df = 1, p-value = 0.0554

On account of the relatively small counts, some statisticians would use Fisher's exact test, which gives P-value $0.061,$ also failing to reject at the 5% level. [See Wikipedia.] This P-value for Fisher's exact test is also shown in Minitab printouts of the pooled and separate variances methods, but was omitted above.

fisher.test(TBL)

    Fisher's Exact Test for Count Data

data:  TBL
p-value = 0.06102
alternative hypothesis: 
 true odds ratio is not equal to 1
95 percent confidence interval:
 0.4375136 1.0319594
sample estimates:
odds ratio 
 0.6730183 
$\endgroup$
1
  • $\begingroup$ Thank you! This was so helpful $\endgroup$ Jun 29, 2021 at 16:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.