# Statistical significance of binomial variable when sample sizes are different?

Suppose we have two groups of students given the same difficult test. However, one group of students was given an extra training session before the test.

normal students = 123

students with extra training = 119

18 of the normal students pass the test, but 25 of the students with extra training pass.

Did the extra training actually make a difference? What are the best tools to use for this? Hypothesis testing? Power analysis?

• Use Fisher's exact test or Pearson Chi-square test. Both of them do not need the sample sizes are the same in two groups. Jun 21, 2019 at 0:41

You're going to want a chi-square test for this.

First, you make a table of your observed values:

        Pass   Fail   Total
Normal  18     105    123
Extra   25     94     119
Total   43     199    242


The numbers in the "Total" row and column are called "marginals" - they represent the total number (out of all 242 students) who passed or failed, or who had normal or extra training.

Next, you calculate percents based on your marginals. 43/242 = overall, 17.8% of students passed. 123/242 = overall, 50.8% of students received extra training. Etc. Then you max a table of expected values, based on these marginals. For example, if the extra training had no effect on passing, then you would expect the same percent of students (17.76%) to pass in both groups. So for how many students you would expect to pass in the "normal" group, you would calculate 242*0.508*0.178 = you expect 21.9 students from the normal group to pass.

Here is the full expected table calculated using that method:

        Pass   Fail   Total
Normal  21.9   101.2  123
Extra   21.1   97.9   119
Total   43     199    242


Now for each cell, calculate (observed - expected)^2/expected, then add that together for all the cells. You should get 0.695 + 0.143 + 0.721 + 0.155 = 1.714.

You are almost ready to look up your test statistic in a chisq table - you just need to know the degrees of freedom. The degrees of freedom is (rows - 1)*(columns - 1) so here you have one degree of freedom.

Now, look up your chi-square test statistic (1.714) in a chi-square table. Find the row for one degree of freedom, then look in the cells for that row. You'll see in the column for p=0.2 the chi-square test statistic is 1.642, and in the column for p=0.1, the chi-square test statistic is 2.706. Your test statistic was in between those two values, so your p-value is somewhere between 0.1 and 0.2. Eg. not significant at the standard p=.05 cutoff.

• Nice (+1). Comparison of your approach and mine. You are essentially doing a two-sided test because of the squaring in the the computation of the chi-squared statistic. The square $(-1.30)^2 = 1.69$ of my Z-statistic is essentially the same is your $\chi^2 = 1.71.$ Your exact P-value is 0.19 and the double of mine is 0.194. (I can see how this could be either a one- or a two-sided test. Although, someone must be hoping that the extra training helps.) Jun 21, 2019 at 1:56
• @BruceET great explanation of the different results! I knew in theory that the chi-square distribution was based on the square of a Z, but this is the first time that's really made sense! Jun 21, 2019 at 4:19

You want to test whether the population rates of passing the exam are the same for the two groups. So your null hypothesis is $$H_0: p_1=p_2$$ vs $$H_a: p_x < p_2.$$

A test of the equality of binomial proportions is discussed in the NIST handbook and implement in Minitab statistical software (among other software programs). Minitab output is as follows:

Test and CI for Two Proportions

Sample   X    N  Sample p
1       18  123  0.146341
2       25  119  0.210084

Difference = p (1) - p (2)
Estimate for difference:  -0.0637426
95% upper bound for difference:  0.0170092
Test for difference = 0 (vs < 0):
Z = -1.30  P-Value = 0.097


This Minitab procedure also includes the P-value for the Fisher Exact Test:

Fisher’s exact test: P-Value = 0.129


While it is true that the students who had extra training showed a somewhat higher pass rate than those who did not (21.0% vs. 14.6%), both P-values exceed 0.05, so at the 5% level of significance that pass rate for the students with additional training did not have a higher pass rate that is statistically significant.

Addendum: Here is the intuitive rationale for Fisher's Exact Test.

In Group 1 and Group 2 combined, you have $$123 + 119 = 242$$ students. Out of the $$18+25=43$$ who passed the exam, only 18 are from Group 1. If all students are equally likely to pass, what is the probability that such a small number of passes in Group 1 would occur at random.

Specifically, let $$X$$ be the number of Group 1 passes out of 43. The P-value of Fisher's Exact Test is $$P(X \le 18).$$

Symbolically, this is $$P(X \le 18) = \sum_{i=1}^{18} \frac{{123 \choose i}{119 \choose 43-i}}{{242 \choose 43}} \approx 0.129,$$

which agrees with Minitab's P-value for the Fisher test.

In R statistical software, this is computed as follows:

 phyper(18, 123, 119, 43)
 0.129473


In the plot of the relevant hypergeometric distribution below, the P-value is the sum of the heights of the bars to the left of the vertical dotted line. 