# Proportion test: one sample or two sample?

We introduced a new way of teaching children this summer. We then split the children (As randomly as possible) into 2 groups. Group 1 children were taught in the new style, Group 2 children were taught in the usual style. (Business as usual). At the end of the summer children in both groups took an exam and we measured what % of children passed that exam.

We want to see if Group 1's children (new instruction method) performed better than Group 2's children (Business as usual).

My question is: should I be conducting a 1 sample test here or a 2 sample test?

In a 2 sample test, I would simply consider the two groups as having received two different treatments (even though group 2 was "business as usual")

In a 1 sample test, I would compare Group 1's data against the pass percentage of Group 2. (think of group 2's pass percentage as fixed)

What would you suggest?

TWO-SAMPLE

TWO-SAMPLE

TWO-SAMPLE

Your group 2 is called a control group. Without a control, you do not know the reason for any changes in performance. Sure, it could be that the new method is more effective. It also could be that you gave a particularly easy exam. Or maybe you keep using the same exam but the questions are getting leaked after a few years.

The control group gives you a baseline measurement against which you can make comparisons.

• Great, thanks. Yep - I know, test vs control. I'd still compare the two. If using the 1 sample, I guess the data being lost is the variance in the control sample. Right? – datapanda Oct 8 at 23:26
• @datapanda Data being lost? I don’t follow your method. – Dave Oct 8 at 23:39
• If you treat the mean of the control group as a fixed constant 'baseline', then you are ignoring variability in the control group. Might be OK if control group had thousands of students and treatment group dozens, but I suppose the two groups are nearly equal in size. // You want a one-sided, two-sample test because you are convinced that the treatment group performed the same as the control group (null) or better (one-sided alternative). – BruceET Oct 8 at 23:48
• Dave - meant data being ignored, sorry. – datapanda Oct 8 at 23:57
• One idea to keep in mind is that you decide on a one-sided or two-sided test (and direction, if one-sides) BEFORE you see the data. If your data show the treatment group to do worse but you decided to do a one-sided test that they did better, you do that test, not a test that they did worse. – Dave Oct 9 at 0:19

Suppose 78 students out of 110 in the control group passed the test and 92 out of 205 in the treatment group passed.

A one-sided two-sample prop.test is appropriate. In R, two different patterns of syntax are possible to present the data. Both are illustrated below.

Both methods result in the same P-value 0.002235, which is smaller than 5%. So you would reject at the 5% level the null hypothesis $$H_0$$ that control and treatments groups have essentially the same pass rate, against the alternative $$H_a$$ that the pass rate for the control group (mentioned first) has a pass rate that is 'less' than the pass rate for the treatment group.

x1 = 78;  n1 = 110;   x2 = 92;  n2 = 105
prop.test(c(78,92), c(110,105), alt="less")

2-sample test for equality of proportions with continuity correction

data:  c(78, 92) out of c(110, 105)
X-squared = 8.0822, df = 1, p-value = 0.002235
alternative hypothesis: less
95 percent confidence interval:
-1.00000000 -0.06908546
sample estimates:
prop 1    prop 2
0.7090909 0.8761905


Alternatively,

x1 = 78;  n1 = 110;   x2 = 92;  n2 = 105
g1 = c(x1, n1-x1);  g2 = c(x2, n2-x2)
TBL = rbind(g1, g2);  TBL   # 'bind' two row vectors to make table
[,1] [,2]
g1   78   32
g2   92   13
prop.test(TBL, alt="less")

2-sample test for equality of proportions with continuity correction

data:  TBL
X-squared = 8.0822, df = 1, p-value = 0.002235
alternative hypothesis: less
95 percent confidence interval:
-1.00000000 -0.06908546
sample estimates:
prop 1    prop 2
0.7090909 0.8761905


Note: If you don't want to use the Yates continuity correction (skipping it is my preference), then use argument cor=F in function prop.test. The continuity correction is biased toward making the chi-squared statistic smaller and the P-value larger. (There is disagreement whether use of the continuity correction is warranted.)

prop.test(TBL, alt="less", cor=F)\$p.val
[1] 0.001303608