1
$\begingroup$

I'm running an analysis on a dataset that captures the test-scores of students before and after they took a class (control group). I also have the same data on an experimental group, test-scores of students before and after they took a class.

I already ran paired t-tests to determine the significance of the mean scores for pre- and post-tests for each group. However, I'd like to look at the difference in percentage change between the two groups. For example, (% change in experimental group) minus (% change in control group), or even the raw score difference of the same note. Is there a test I can run to determine if the difference in differences is statistically significant?

Thank you so much!

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ There’s a method called difference in differences: en.wikipedia.org/wiki/Difference_in_differences. $\endgroup$
    – Dave
    Commented Jun 28, 2020 at 16:12
  • $\begingroup$ Caution: If you simulate a simple regression model and are pleased with the accuracy of the coefficients, and then change the model to a difference in the Ys vs. difference in Xs, now just lost one data point, but the error variance is doubled! As such, what was a good estimate of the slope is now evidently worst. Performing yet another difference would further augment the error structure. Advice: Avoid this path if possible. If you have any doubts on my claims, it is easy to verify. $\endgroup$
    – AJKOER
    Commented Jun 28, 2020 at 16:26

1 Answer 1

Reset to default
1
$\begingroup$

This is a simple difference in difference model which can be handled via linear regression in most cases. Here, I've simulated some data. The baseline grades for both groups are 75. The control group experiences a 2 point increase after taking the class and the treatment group experiences a 3 point increase (2 points from the class, 1 for being in the treatment group.

set.seed(0)
N = 100
id = rep(1:N,2)
time = sort(rep(0:1,N))
txt = rep(rbinom(N,1,0.5),2)
y = 75 + 0*txt + 2*time + 1*txt*time + rnorm(2*N, 0, 5)

This can be analyzed via the lm command

d = tibble(y,txt,time, id)

model = lm(y~txt*time, data = d)

The results from the model are

Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 74.57405    0.67298 110.812   <2e-16 ***
txt         -0.09951    0.97136  -0.102   0.9185    
time         2.35543    0.95173   2.475   0.0142 *  
txt:time     2.11074    1.37371   1.537   0.1260    
---

The model estimates the intercept correctly, along with the effect of time. However, the effect of time in the treatment group is not significant, even though the coefficient in the simulation was. What gives? An interaction for binary covariates will require much more data to achieve the same power as analyzing a single binary covariate. In any case, this does not make the analysis invalid, only under powered.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Yes, I agree with first part of the statement " In any case, this does not make the analysis invalid, only under powered." but not the last. Better would be caution that the difference model is potentially more noisy, so is this approach warranted? $\endgroup$
    – AJKOER
    Commented Jun 28, 2020 at 17:32
  • $\begingroup$ @AJKOER It isn't clear from either of your comments what you're talking about. I would encourage you to post an answer addressing any issues you see as salient rather than remaining in the comments and asking questions without contributing proposed ways forward. $\endgroup$
    – Demetri Pananos
    Commented Jun 28, 2020 at 17:56
  • $\begingroup$ First comment cites a doubling in error variance as a consequence of taking a difference between successive points in a simple linear regression. My actual study of such a scenario confirms this and suggests a degradation in the 'power'. I would ask you to do this exercise to confirm, but your work above and very words are confirming. My essential point remains, why look at a differencing analysis if not required. $\endgroup$
    – AJKOER
    Commented Jun 28, 2020 at 19:55
  • $\begingroup$ Perhaps some math helps. Upon differencing, Y2 - Y1 = b*(X2 - X1) + e2 - e1. The expected variance of the independent error terms for this new difference model (followed by Y3-Y2, etc) is now Var(e1 - e1) = Var(e1) + Var(e2) = 2 Sigma square. $\endgroup$
    – AJKOER
    Commented Jun 28, 2020 at 20:06
  • $\begingroup$ @AJKOER I think it would be best if you posted your own answer, with code, to demonstrate your point. You can use mine as a starting point. $\endgroup$
    – Demetri Pananos
    Commented Jun 28, 2020 at 21:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.