Statistical significance of binomial variable when sample sizes are different?

Question

Suppose we have two groups of students given the same difficult test. However, one group of students was given an extra training session before the test.

normal students = 123

students with extra training = 119

18 of the normal students pass the test, but 25 of the students with extra training pass.

Did the extra training actually make a difference? What are the best tools to use for this? Hypothesis testing? Power analysis?

Answer 1

You're going to want a chi-square test for this.

First, you make a table of your observed values:

        Pass   Fail   Total
Normal  18     105    123
Extra   25     94     119
Total   43     199    242

The numbers in the "Total" row and column are called "marginals" - they represent the total number (out of all 242 students) who passed or failed, or who had normal or extra training.

Next, you calculate percents based on your marginals. 43/242 = overall, 17.8% of students passed. 123/242 = overall, 50.8% of students received extra training. Etc. Then you max a table of expected values, based on these marginals. For example, if the extra training had no effect on passing, then you would expect the same percent of students (17.76%) to pass in both groups. So for how many students you would expect to pass in the "normal" group, you would calculate 242*0.508*0.178 = you expect 21.9 students from the normal group to pass.

Here is the full expected table calculated using that method:

        Pass   Fail   Total
Normal  21.9   101.2  123
Extra   21.1   97.9   119
Total   43     199    242

Now for each cell, calculate (observed - expected)^2/expected, then add that together for all the cells. You should get 0.695 + 0.143 + 0.721 + 0.155 = 1.714.

You are almost ready to look up your test statistic in a chisq table - you just need to know the degrees of freedom. The degrees of freedom is (rows - 1)*(columns - 1) so here you have one degree of freedom.

Now, look up your chi-square test statistic (1.714) in a chi-square table. Find the row for one degree of freedom, then look in the cells for that row. You'll see in the column for p=0.2 the chi-square test statistic is 1.642, and in the column for p=0.1, the chi-square test statistic is 2.706. Your test statistic was in between those two values, so your p-value is somewhere between 0.1 and 0.2. Eg. not significant at the standard p=.05 cutoff.

Answer 2

You want to test whether the population rates of passing the exam are the same for the two groups. So your null hypothesis is $H_0: p_1=p_2$ vs $H_a: p_x < p_2.$

A test of the equality of binomial proportions is discussed in the NIST handbook and implement in Minitab statistical software (among other software programs). Minitab output is as follows:

Test and CI for Two Proportions 

Sample   X    N  Sample p
1       18  123  0.146341
2       25  119  0.210084

Difference = p (1) - p (2)
Estimate for difference:  -0.0637426
95% upper bound for difference:  0.0170092
Test for difference = 0 (vs < 0):  
    Z = -1.30  P-Value = 0.097

This Minitab procedure also includes the P-value for the Fisher Exact Test:

Fisher’s exact test: P-Value = 0.129

While it is true that the students who had extra training showed a somewhat higher pass rate than those who did not (21.0% vs. 14.6%), both P-values exceed 0.05, so at the 5% level of significance that pass rate for the students with additional training did not have a higher pass rate that is statistically significant.

Addendum: Here is the intuitive rationale for Fisher's Exact Test.

In Group 1 and Group 2 combined, you have $123 + 119 = 242$ students. Out of the $18+25=43$ who passed the exam, only 18 are from Group 1. If all students are equally likely to pass, what is the probability that such a small number of passes in Group 1 would occur at random.

Specifically, let $X$ be the number of Group 1 passes out of 43. The P-value of Fisher's Exact Test is $P(X \le 18).$

Symbolically, this is $$P(X \le 18) = \sum_{i=1}^{18} \frac{{123 \choose i}{119 \choose 43-i}}{{242 \choose 43}} \approx 0.129,$$

which agrees with Minitab's P-value for the Fisher test.

In R statistical software, this is computed as follows:

 phyper(18, 123, 119, 43)
[1] 0.129473

In the plot of the relevant hypergeometric distribution below, the P-value is the sum of the heights of the bars to the left of the vertical dotted line.