I have three groups of data, each with a binomial distribution (i.e. each group has elements that are either success or failure). I do not have a predicted probability of success, but instead can only rely on the success rate of each as an approximation for the true success rate. I have only found this question, which is close but does not seem to exactly deal with the this scenario.

To simplify down the test, let's just say that I have 2 groups (3 can be extended from this base case).

  • Group 1 trials: $n_1$ = 2455
  • Group 2 trials: $n_2$ = 2730

  • Group 1 success: $k_1$ = 1556
  • Group 2 success: $k_2$ = 1671

I don't have an expected success probability, only what I know from the samples. So my implied success rate for the two groups is:

  • Group 1 success rate: $p_1$ = 1556/2455 = 63.4%
  • Group 2 success rate: $p_2$ = 1671/2730 = 61.2%

The success rate of each of the sample is fairly close. However my sample sizes are also quite large. If I check the CDF of the binomial distribution to see how different it is from the first (where I'm assuming the first is the null test) I get a very small probability that the second could be achieved.

In Excel:

1-BINOM.DIST(1556,2455,61.2%,TRUE) = 0.012

However, this does not take into account any variance of the first result, it just assumes the first result is the test probability.

Is there a better way to test if these two samples of data are actually statistically different from one another?

  • $\begingroup$ Another question I came across that didn't really help much: stats.stackexchange.com/questions/82059/… $\endgroup$ – Scott Aug 28 '14 at 17:15
  • $\begingroup$ Does this question help? stats.stackexchange.com/questions/25299/… $\endgroup$ – Eric Aug 28 '14 at 17:32
  • 2
    $\begingroup$ In R, you could use prop.test: prop.test(c(1556, 1671), c(2455, 2730)). $\endgroup$ – COOLSerdash Aug 28 '14 at 17:48
  • 1
    $\begingroup$ Could be done as a two-sample (binomial) proportions test, or a 2x2 chi-square $\endgroup$ – Glen_b -Reinstate Monica Aug 28 '14 at 17:58
  • 1
    $\begingroup$ Extending the base case from two groups to three could be problematic, because the tests will be interdependent: you will need a binomial version of ANOVA to handle that. $\endgroup$ – whuber Mar 21 '16 at 1:42

The solution is a simple google away: http://en.wikipedia.org/wiki/Statistical_hypothesis_testing

So you would like to test the following null hypothesis against the given alternative

$H_0:p_1=p_2$ versus $H_A:p_1\neq p_2$

So you just need to calculate the test statistic which is

$$z=\frac{\hat p_1-\hat p_2}{\sqrt{\hat p(1-\hat p)\left(\frac{1}{n_1}+\frac{1}{n_2}\right)}}$$

where $\hat p=\frac{n_1\hat p_1+n_2\hat p_2}{n_1+n_2}$.

So now, in your problem, $\hat p_1=.634$, $\hat p_2=.612$, $n_1=2455$ and $n_2=2730.$

Once you calculate the test statistic, you just need to calculate the corresponding critical region value to compare your test statistic too. For example, if you are testing this hypothesis at the 95% confidence level then you need to compare your test statistic against the critical region value of $z_{\alpha/2}=1.96$ (for this two tailed test).

Now, if $z>z_{\alpha/2}$ then you may reject the null hypothesis, otherwise you must fail to reject the null hypothesis.

Well this solution works for the case when you are comparing two groups, but it does not generalize to the case where you want to compare 3 groups.

You could however use a Chi Squared test to test if all three groups have equal proportions as suggested by @Eric in his comment above: " Does this question help? stats.stackexchange.com/questions/25299/ … – Eric"

  • 6
    $\begingroup$ Thanks @Dan. As many times with Google, knowing the right term to search for is the first hurdle. I did take a look at the chi-squared test. The problem there, as with where I was first getting stuck, is that my expected calculation is based on the sample. I can't therefore provide an expected value, because my samples are used to determine that expected value. $\endgroup$ – Scott Aug 28 '14 at 18:33
  • $\begingroup$ @Scott, if your hypothesized proportions for the three groups are that they are all equal then the expected value should be 1/3 for each group. $\endgroup$ – Dan Aug 28 '14 at 18:35
  • 1
    $\begingroup$ A related explanation of using this test can be found here: itl.nist.gov/div898/handbook/prc/section3/prc33.htm (currently, the Wikipedia page does not provide a walk-through example). $\endgroup$ – wwwilliam Apr 18 '17 at 3:29
  • $\begingroup$ Can someone help me prove the standard deviation of the difference between the two binomial distributions, in other words prove that : $$\sqrt{\hat p (1-\hat p)(\frac{1}{n_1} + \frac{1}{n_2})} = \sqrt{\frac{\hat p_1 (1-\hat p_1)}{n_1} + \frac{\hat p_2 (1-\hat p_2)}{n_2}}$$ $\endgroup$ – Tanguy Aug 4 '18 at 9:36
  • $\begingroup$ answer to my question can be found here: stats.stackexchange.com/questions/361015/… $\endgroup$ – Tanguy Aug 8 '18 at 10:22

In R the answer is calculated as:

fisher.test(rbind(c(1556,2455-1556), c(1671,2730-1671)), alternative="less")
  • 8
    $\begingroup$ Would you consider writing a little bit more than providing the R function? Naming the function does not help in understanding the problem and not everyone use R, so it would be no help for them. $\endgroup$ – Tim Dec 8 '14 at 13:52
  • 1
    $\begingroup$ This is the most exact statistical answer, and works for small numbers of observations (see the following: itl.nist.gov/div898/handbook/prc/section3/prc33.htm). $\endgroup$ – Andrew Mao Mar 30 '15 at 19:36
  • 1
    $\begingroup$ Fishers exact test en.wikipedia.org/wiki/Fisher's_exact_test $\endgroup$ – Keith May 22 '15 at 23:59

Just a summary:

Dan and Abaumann's answers suggest testing under a binomial model where the null hypothesis is a unified single binomial model with its mean estimated from the empirical data. Their answers are correct in theory but they need approximation using normal distribution since the distribution of test statistic does not exactly follow Normal distribution. Therefore, it's only correct for a large sample size.

But David's answer is indicating a nonparametric test using Fisher's test.The information is here: https://en.wikipedia.org/wiki/Fisher%27s_exact_test And it can be applied to small sample sizes but hard to calculate for big sample sizes.

Which test to use and how much you trust your p-value is a mystery. But there are always biases in whichever test to choose.

  • 2
    $\begingroup$ Are you trying to suggest that sample sizes in the thousands, with likely parameter values near $1/2$, are not large for this purpose? $\endgroup$ – whuber May 3 '16 at 1:03
  • 1
    $\begingroup$ For this case, I think you could use Dan's method but compute the p value in an exact way (binomial) and approxiamte way (normal Z>Φ−1(1−α/2)Z>Φ−1(1−α/2) and Z<Φ−1(α/2) ) to compare whether they are close enough. $\endgroup$ – Dr_Hope May 6 '16 at 4:39

Your test statistic is $Z = \frac{\hat{p_1}-\hat{p_2}}{\sqrt{\hat{p}(1-\hat{p})(1/n_1+1/n_2)}}$, where $\hat{p}=\frac{n_1\hat{p_1}+n_2\hat{p_2}}{n_1+n_2}$.

The critical regions are $Z > \Phi^{-1}(1-\alpha/2)$ and $Z<\Phi^{-1}(\alpha/2)$ for the two-tailed test with the usual adjustments for a one-tailed test.


In Python, statsmodels has a function called proportions_ztest. Here is an example of its usage:

import statsmodels.api as sm
import numpy as np
import rpy2.robjects.packages as rpackages
import rpy2.robjects as robjects
rstats = rpackages.importr('stats')

s1 = 1556
n1 = 2455

s2 = 1671
n2 = 2730

# manual calculation
p1 = s1 / n1
p2 = s2 / n2
p = (s1 + s2) / (n1 + n2)

z = (p1 - p2) / (p*(1-p)*((1/n1)+(1/n2)))**0.5

# using R in Python with rpy2
rmatrix = robjects.r.matrix(robjects.IntVector([s1, n1-s1, s2,n2-s2]), nrow=2)
fisher_test = rstats.fisher_test(rmatrix, alternative="two.sided")

zscore, pval = sm.stats.proportions_ztest([s1, s2], [n1, n2], alternative='two-sided')

print('Manual calculation of z: {:.6f}'.format(z))
print('Z-score from statsmodels: {:.6f}'.format(zscore))
print('R pvalue from fisher.test: {:.6f}'.format(fisher_test[0][0]))
print('Statsmodels pvalue: {:.6f}'.format(pval))

This prints out:

Manual calculation of z: 1.610825
Z-score from statsmodels: 1.610825
R pvalue from fisher.test: 0.108268
Statsmodels pvalue: 0.107218

Original post: Dan's answer is actually incorrect, not to offend anyone. A z-test is used only if your data follows a standard normal distribution. In this case, your data follows a binomial distribution, therefore a use a chi-squared test if your sample is large or fisher's test if your sample is small.

Edit: My mistake, apologies to @Dan. A z-test is valid here if your variables are independent. If this assumption is not met or unknown, a z-test may be invalid.

  • 3
    $\begingroup$ The "only if" part is an extreme position unlikely to be shared by many. No data actually follow a normal distribution. Few data actually behave as if drawn randomly and independently from a normal distribution. Nevertheless, z tests continue to be effective because the distributions of statistics (such as the difference of means) to which they apply can be extremely well approximated by normal distributions. In fact, the appeal to a $\chi^2$ test relies on the same asymptotic assumptions as a z test does! $\endgroup$ – whuber Mar 21 '16 at 1:44
  • $\begingroup$ If you believe in the CLT, then the normal distribution does commonly exist. $\endgroup$ – Ryan Mar 21 '16 at 2:49
  • 2
    $\begingroup$ @Ryan Well, I believe in the CLT but it doesn't say anything about n=30 or n=300 or n=5000. You don't actually get normality unless you somehow manage to have infinite sample sizes, or you somehow started with normality. Questions about how close we are to normality when taking averages are not addressed by the CLT.. (We can consider those questions but we don't use the CLT to find out if the approximation is any good.) $\endgroup$ – Glen_b -Reinstate Monica Jul 26 '16 at 5:12

Not the answer you're looking for? Browse other questions tagged or ask your own question.