Significance Test for the Delta of Deltas I'm running an analysis on a dataset that captures the test-scores of students before and after they took a class (control group). I also have the same data on an experimental group, test-scores of students before and after they took a class.
I already ran paired t-tests to determine the significance of the mean scores for pre- and post-tests for each group. However, I'd like to look at the difference in percentage change between the two groups. For example, (% change in experimental group) minus (% change in control group), or even the raw score difference of the same note. Is there a test I can run to determine if the difference in differences is statistically significant?
Thank you so much!
 A: This is a simple difference in difference model which can be handled via linear regression in most cases.  Here, I've simulated some data.  The baseline grades for both groups are 75.  The control group experiences a 2 point increase after taking the class and the treatment group experiences a 3 point increase (2 points from the class, 1 for being in the treatment group.
set.seed(0)
N = 100
id = rep(1:N,2)
time = sort(rep(0:1,N))
txt = rep(rbinom(N,1,0.5),2)
y = 75 + 0*txt + 2*time + 1*txt*time + rnorm(2*N, 0, 5)

This can be analyzed via the lm command
d = tibble(y,txt,time, id)

model = lm(y~txt*time, data = d)


The results from the model are
Coefficients:
            Estimate Std. Error t value Pr(>|t|)    
(Intercept) 74.57405    0.67298 110.812   <2e-16 ***
txt         -0.09951    0.97136  -0.102   0.9185    
time         2.35543    0.95173   2.475   0.0142 *  
txt:time     2.11074    1.37371   1.537   0.1260    
---


The model estimates the intercept correctly, along with the effect of time.  However, the effect of time in the treatment group is not significant, even though the coefficient in the simulation was.  What gives? An interaction for binary covariates will require much more data to achieve the same power as analyzing a single binary covariate.  In any case, this does not make the analysis invalid, only under powered.
