# diff-in-diff for binary outcome? (i.e.: Comparing proportions of two groups over two time points in paired setting)

Let's say I have students randomly split into two groups (treatment and control). Each student is given a test and they either pass or fail. They then either take a course (the students in "treatment") or they don't (students in "control") - then they take the exam again.

I want to know if the course helped the students who took the exam in the second time.

If we had only the second time, and wanted to compare treatment to control, we could use chi-square test (or fisher's exact test).

If we had only one group (e.g.: treatment), and we wanted to compare the success in the example before-after the course, we could have used mcnamar test.

But what test would I use to compare the effect of the course on the improvement in the exam's pass rates?

If the outcome wasn't pass/fail, but a normal outcome, we could have taken the diff between the before and after for each student, and then compare the two groups using a two sample t-test (i.e.: diff-in-diff).

But would should be done with a binary outcome?

• Do those who "pass" take the exam again? Do they also possibly take the course again if assigned to "treatment"? Is this actually a randomized design? Mar 1 at 18:01
• It's helpful to think of these models in a regression setting. Why not use covariates equal to (control or treatment) and (pass or not first time) with response (pass or not second time)? Mar 1 at 18:01

I have no doubt that an appropriate regression model might be more elegant and might allow for testing more details than can be done with a couple of t tests. However, it seems to me that the two most important issues are (a) whether the treatment group improved from the first exam to the second, possibly on account of learning something in the course. and (b) whether the treatment group improved more than the control group from the first to the second exam.

If you have about 30 students in each group and the exams have about 50 T-F questions each, then the scores might be nearly normal so that (a) could be answered with a paired t test (or a one-sample t test on differences, and (b) could be answered with a two-sample t test. With such sample sizes and no indications of serious departures from normality, my first analyses would be to do these two tests to get an idea whether the major expectations of the study are met.

I have simulated some fake data for purposes of illustration. Let the improvement scores for treatment and control groups be td and cd, respectively.

summary(td)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
5.00    7.25   10.00    9.40   11.00   14.00
summary(cd)
Min. 1st Qu.  Median    Mean 3rd Qu.    Max.
0.000   3.000   4.000   4.367   5.750   9.000


For my fake data, boxplots (Treatment=1) show no serious skewness or outliers.

boxplot(td, cd, horizontal=T)


Normal probability plots show discreteness but no serious departures for normality.

par(mfrow=c(1,2))
qqnorm(td,main="Treatment: Normal Probability Plot"); qqline(td)
qqnorm(cd,main="Control: Normal Probability Plot"); qqline(cd)
par(mfrow=c(1,1))


Thus, it seems reasonable to compare 'improvement' scores with a two-sample t test. The P-value is very nearly $$0$$ and we reject the null hypothesis (of no difference).

t.test(td, cd)

Welch Two Sample t-test

data:  td and cd
t = 8.5125, df = 57.587, p-value = 9.006e-12
alternative hypothesis: true difference in means is not equal to 0
95 percent confidence interval:
3.849569 6.217098
sample estimates:
mean of x mean of y
9.400000  4.366667


Also, a right-sided, one-sample t test on the improvement scores in the treatment group, finds statistically significant improvement.

t.test(td, alt="gr")

One Sample t-test

data:  td
t = 23.5, df = 29, p-value < 2.2e-16
alternative hypothesis: true mean is greater than 0
95 percent confidence interval:
8.720349      Inf
sample estimates:
mean of x
9.4


Addendum: In your Question, it seems that you might not have exam scores; just whether subjects passed or failed the exams. What's above was written hoping you do have scores. But even if not, I suppose you know, whether a subject improved (went from fail to pass), stayed the same, or went from pass to fail.

Then for Treatment subjects you might have 3 down, 5 same, and 22 up. You could do a proportion test to see if the proportion who improved is significantly better than half. With these counts the answer is Yes. [These hypothetical counts are not derived from the simulated data used for the t tests.]

prop.test(c(22, 8), c(30,30), alt="gr", cor=F)

2-sample test for equality of proportions
without continuity correction

data:  c(22, 8) out of c(30, 30)
X-squared = 13.067, df = 1, p-value = 0.0001503
alternative hypothesis: greater
95 percent confidence interval:
0.2788575 1.0000000
sample estimates:
prop 1    prop 2
0.7333333 0.2666667


By contrast, if the Control group had 4 down, 21 same, 5, improved, then you could do a chi-squared test to see if the Treatment group did significantly better. The result is that Treatment and Control groups are clearly not homogeneous. (The simulation option was used because the cells for subjects going from pass to fail have so few counts.)

MAT = cbind(c(3, 5, 22), c(4, 21, 5))
chisq.test(MAT, sim=T)

Pearson's Chi-squared test with simulated p-value
(based on 2000 replicates)

data:  MAT
X-squared = 20.693, df = NA, p-value = 0.0004998


Fisher's exact test is another option, also with a highly significant result.

fisher.test(MAT)

Fisher's Exact Test for Count Data

data:  MAT
p-value = 1.123e-05
alternative hypothesis: two.sided


Note: The data for t tests were sampled in R according to the program below.

set.seed(301)
t1 = rbinom(30, 50, .5)
td = rbinom(30, 50, .2)
t2 = t1 + td
c1 = rbinom(30, 50, .5)
cd = rbinom(30, 50, .1)-1
c2 = c1 + cd