I want to test if a certain pill would make any difference in avoiding the flu.

I give the pill to group A and a placebo to group B. Then I count how many individuals got the flu in each group, and how many didn't.

I imagine two ways of testing if the pill made any difference in avoiding the flu:

Z test for proportion: I conduct a z test with the following hypoteshis: Null hypothesis: The proportion of people that got the flu in group A is equal to the proportion of people that got the flu in group B. Alternative Hypothesis: the proportions are different.

Chi-squared test: I create a contingency table with the sample and I conduct a chi-squared test for homogeneity.

What's the most adequate test in this case, and why?


There are various versions of both tests. If the error variance in the test of proportions is estimated from the two groups separately, and no continuity correction is used for either test, then the two tests may be equivalent.

Suppose there were 403 successes out of 500 subjects in the drug group and 366 out of 500 in the placebo group, then here are the two tests as computed in R. Notice that the methods of formatting the data and the methods of formatting the output differ between the two tests as implemented in R.

Test of binomial proportions. Data are numbers of successes and sample sizes. The continuity correction is declined (parameter cor=F).

prop.test(c(403,366), c(500,500), cor=F)

    2-sample test for equality of proportions 
    without continuity correction

data:  c(403, 366) out of c(500, 500)
X-squared = 7.7066, df = 1, p-value = 0.005502
alternative hypothesis: two.sided
95 percent confidence interval:
 0.0219564 0.1260436
sample estimates:
prop 1 prop 2 
 0.806  0.732 

Chi-squared test. Data are provided in a contingency table. The Yates correction is declined.

     [,1] [,2]
[1,]  403  366
[2,]   97  134

chisq.test(TBL, cor=F)

        Pearson's Chi-squared test

data:  TBL
X-squared = 7.7066, df = 1, p-value = 0.005502

Notes: (1) Proportions test: If a pooled error variance is used in the test of binomial proportions, then the confidence intervals do not exactly match the significance test. That is, 95% confidence intervals excluding $0$ do not exactly match rejection at the 5% level. Also, the X-squared statistic shown in the R output for this test is the square of the z-statistic, shown by some other statistical software. Relevant Minitab output below:

Test and CI for Two Proportions 

Sample    X    N  Sample p
1       403  500  0.806000
2       366  500  0.732000

Difference = p (1) - p (2)
Estimate for difference:  0.074
95% CI for difference:  (0.0219564, 0.126044)
Test for difference = 0 (vs ≠ 0):  Z = 2.79  P-Value = 0.005

Fisher’s exact test: P-Value = 0.007

(2) Chi-squared test: If counts are sufficiently small for the (somewhat overly conservative) Yates correction to make a difference whether to reject, then perhaps it is best to use a Fisher exact test instead. The P-value of Fisher's test for my fake data is shown below.

[1] 0.006834361

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.