I have a dataset where the issue is of this form.

There are two groups, Group A (N=5000) and Group B (N=1000). Let's say 5 people in Group A develop a certain disease, and only 2 in group B do.

Then the proportion for A is 5/5000 -> 0.001 and for B it is 2/1000 -> 0.002.

How can I test if the proportion between these two groups is statistically significant?

The tests I found online rely on the Central Limit Theorem, such that np>=10 and n(1-p) >= 10, which does not hold for my dataset. Are there any other approaches?

  • $\begingroup$ Use Fisher Exact Test per discussion in Answer. $\endgroup$
    – BruceET
    Apr 22 '19 at 2:38

The traditional approach would be to do a 2-sample test for a difference in proportions: In Minitab, results of this test are as shown below. The warning about the first P-value from a normal approximation causes doubt, roughly for the reasons you mention.

However, the result from Fisher's exact test uses an exact hypergeometric probability. It also shows no significant difference.

Test and CI for Two Proportions 

Sample  X     N  Sample p
1       5  5000  0.001000
2       2  1000  0.002000

Difference = p (1) - p (2)
Estimate for difference:  -0.001
95% upper bound for difference:  0.00143738
Test for difference = 0 (vs < 0):  
  Z = -0.67  P-Value = 0.250

* NOTE * The normal approximation may be 
inaccurate for small samples.

Fisher’s exact test: P-Value = 0.330

A direct hypergeometric computation in R can be argued as follows. Suppose an urn contains $5000$ tokens marked A and $1000$ marked B. Seven tokens are taken at random without replacement, corresponding to disease. What is the probability five or fewer of those are marked A?

The answer is

$$\sum_{k=0}^5\frac{{5000 \choose k}{1000 \choose 7-k}}{{6000 \choose 7}} = 0.3302,$$

which agrees with the P-value from Fisher's exact test.

In R, the computation can be done in terms of a hypergeometric CDF:

phyper(5, 5000, 1000, 7)
[1] 0.330204

Here is a plot of this hypergeometric distribution. The P-value is the sum of the heights of the bars to the left of the vertical dotted line.

enter image description here


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.