1
$\begingroup$

We introduced a new way of teaching children this summer. We then split the children (As randomly as possible) into 2 groups. Group 1 children were taught in the new style, Group 2 children were taught in the usual style. (Business as usual). At the end of the summer children in both groups took an exam and we measured what % of children passed that exam.

We want to see if Group 1's children (new instruction method) performed better than Group 2's children (Business as usual).

My question is: should I be conducting a 1 sample test here or a 2 sample test?

In a 2 sample test, I would simply consider the two groups as having received two different treatments (even though group 2 was "business as usual")

In a 1 sample test, I would compare Group 1's data against the pass percentage of Group 2. (think of group 2's pass percentage as fixed)

What would you suggest?

$\endgroup$
0
$\begingroup$

TWO-SAMPLE

TWO-SAMPLE

TWO-SAMPLE

Your group 2 is called a control group. Without a control, you do not know the reason for any changes in performance. Sure, it could be that the new method is more effective. It also could be that you gave a particularly easy exam. Or maybe you keep using the same exam but the questions are getting leaked after a few years.

The control group gives you a baseline measurement against which you can make comparisons.

$\endgroup$
  • $\begingroup$ Great, thanks. Yep - I know, test vs control. I'd still compare the two. If using the 1 sample, I guess the data being lost is the variance in the control sample. Right? $\endgroup$ – datapanda Oct 8 at 23:26
  • $\begingroup$ @datapanda Data being lost? I don’t follow your method. $\endgroup$ – Dave Oct 8 at 23:39
  • $\begingroup$ If you treat the mean of the control group as a fixed constant 'baseline', then you are ignoring variability in the control group. Might be OK if control group had thousands of students and treatment group dozens, but I suppose the two groups are nearly equal in size. // You want a one-sided, two-sample test because you are convinced that the treatment group performed the same as the control group (null) or better (one-sided alternative). $\endgroup$ – BruceET Oct 8 at 23:48
  • $\begingroup$ Dave - meant data being ignored, sorry. $\endgroup$ – datapanda Oct 8 at 23:57
  • 1
    $\begingroup$ One idea to keep in mind is that you decide on a one-sided or two-sided test (and direction, if one-sides) BEFORE you see the data. If your data show the treatment group to do worse but you decided to do a one-sided test that they did better, you do that test, not a test that they did worse. $\endgroup$ – Dave Oct 9 at 0:19
0
$\begingroup$

Suppose 78 students out of 110 in the control group passed the test and 92 out of 205 in the treatment group passed.

A one-sided two-sample prop.test is appropriate. In R, two different patterns of syntax are possible to present the data. Both are illustrated below.

Both methods result in the same P-value 0.002235, which is smaller than 5%. So you would reject at the 5% level the null hypothesis $H_0$ that control and treatments groups have essentially the same pass rate, against the alternative $H_a$ that the pass rate for the control group (mentioned first) has a pass rate that is 'less' than the pass rate for the treatment group.

x1 = 78;  n1 = 110;   x2 = 92;  n2 = 105
prop.test(c(78,92), c(110,105), alt="less")

    2-sample test for equality of proportions with continuity correction

data:  c(78, 92) out of c(110, 105)
X-squared = 8.0822, df = 1, p-value = 0.002235
alternative hypothesis: less
95 percent confidence interval:
 -1.00000000 -0.06908546
sample estimates:
   prop 1    prop 2 
0.7090909 0.8761905 

Alternatively,

x1 = 78;  n1 = 110;   x2 = 92;  n2 = 105
g1 = c(x1, n1-x1);  g2 = c(x2, n2-x2)
TBL = rbind(g1, g2);  TBL   # 'bind' two row vectors to make table
   [,1] [,2]
g1   78   32
g2   92   13
prop.test(TBL, alt="less")

    2-sample test for equality of proportions with continuity correction

data:  TBL
X-squared = 8.0822, df = 1, p-value = 0.002235
alternative hypothesis: less
95 percent confidence interval:
 -1.00000000 -0.06908546
sample estimates:
   prop 1    prop 2 
0.7090909 0.8761905 

Note: If you don't want to use the Yates continuity correction (skipping it is my preference), then use argument cor=F in function prop.test. The continuity correction is biased toward making the chi-squared statistic smaller and the P-value larger. (There is disagreement whether use of the continuity correction is warranted.)

prop.test(TBL, alt="less", cor=F)$p.val 
[1] 0.001303608
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.