Does Differences-in-Differences linear regression 'forget' the pairing between before and after data points?

Question

Differences-in-differences regression can be used to test the impact of a treatment on a metric of interest. It works by comparing the metric before and after, both for a treatment group and for a control group. For example, I might want to measure whether an educational intervention improves students' grades. To do this, I would measure grades before and after the intervention, both for the participating group and for a control group (who didn't participate).

The input data might have the following form:

Student	Treated/Control	Before grade (%)	After grade (%)
Student A	Treated	40	60
Student B	Treated	45	55
Student C	Control	30	50
Student D	Treated	75	80
$\vdots$	$\vdots$	$\vdots$	$\vdots$

If I understand correctly, the linear model underlying the differences-in differences-method is as follows:

$Y_i=\alpha+\beta T_i+\gamma A_i+\delta T_i A_i+\epsilon_i$,

where $Y_i$ is the $i^\textrm{th}$ grade, $T_i$ is a dummy variable which takes the value 1 if the $i^\textrm{th}$ data point was from the treated group, $A_i$ is a dummy variable which takes the value 1 if the $i^\textrm{th}$ data point was taken after the intervention, and $\epsilon_i$ is a normal random variable with mean 0. The Greek letters are parameters to be estimated. If we want to test whether the treatment has an effect, we will test the null hypothesis $\delta=0$ (because $\delta$ tells us how much extra the treatment group gained from before to after vs the control group).

The above data doesn't match the form of this model. To use the linear model, we need to have only one grade measurement per row, and to introduce variable telling us whether each data point was before or after.

Student	Treated? ($T_i$)	After? ($A_i$)	Grade ($Y_i$)
Student A	1	0	40
Student A	1	1	60
Student B	1	0	45
Student B	1	1	55
Student C	0	0	30
Student C	0	1	50
$\vdots$	$\vdots$	$\vdots$	$\vdots$

I understand how to run the linear regression on data in this form.

---

The question

My question is whether reformatting this data was the right thing to do. In particular, by running the analysis on this data, we have lost the information that the first two data points belonged to Student A. The data below would give equivalent results - the after grades of Student A and Student B have been swapped.

Student	Treated? ($T_i$)	After? ($A_i$)	Grade ($Y_i$)
Student A	1	0	40
Student A	1	1	55$\leftarrow$
Student B	1	0	45
Student B	1	1	60$\leftarrow$
Student C	0	0	30
Student C	0	1	50
$\vdots$	$\vdots$	$\vdots$	$\vdots$

Intuitively, it seems that this information was important. On the other hand, perhaps it is not: by using a linear model (and holding all the information about the model in the four Greek variables), we have produced a set up where four means (before/after for treatment/control) are a set of sufficient statistics for the model. Switching the final grades of Student A and Student B doesn't change these means, so perhaps the two situations should be regarded as equivalent.

Answer 1

The first data frame includes multiple $i$ students and their associated test scores measured at two distinct time periods $t$. The before-and-after scores (i.e., pre- versus post-test) are juxtaposed side-by-side, giving the data frame a wide display. You then propose the following model which I reproduced below:

$$ Y_i = \alpha + \beta T_i + \gamma A_i + \delta (T_i \times A_i) + \epsilon_i, $$

which includes the appropriate variables but the wrong subscripts! For instance, every variable is $i$-subscripted, indicating you observe individual grades in one time period. $T_i$ is appropriately subscripted; it should index students in your treatment group, 0 otherwise. $A_i$, on the other hand, is devoid of a $t$-subscript despite denoting the post-intervention period. It should equal 1 in the "after" period in both treatment and control groups. The $i$-subscript appended to $A$ does not account for the simple passage of time between tests. If applied to your first data frame which only includes one observation per student, then a linear model run in standard software cannot disambiguate between pre- and post-test grades unless it is stacked. It will not return a difference-in-differences estimate.

Here is the appropriate model, which is estimable if used on your second (stacked) data frame:

$$ Y_{it} = \alpha + \beta T_i + \gamma A_t + \delta (T_i \times A_t) + \epsilon_{it}, $$

where you observe students $i$ in time periods $t$. By stacking observations, you do not lose the second time period as your question suggests. $A_t$ is a column vector and distinguishes between pre- and post-test values. In most software packages, the data must be formatted in this manner (i.e., long format). I hope it is obvious how the wide display might become unwieldy in settings with serial observations before-and-after treatment. Suppose you sample multiple $i$ students at a university and observe a series of outcomes for each student over 100 consecutive days. Now suppose you also adjusted for a series of time-varying covariates at the student level. Would you organize your data in a wide display in this setting? In my opinion, the stacked display is the way to go.

Even though I don't particularly advise it, you can certainly structure your data where each row corresponds to one student. In the classical case with two discrete groups (i.e., treatment group versus control group) and two discrete time periods (i.e., pre-treatment versus post-treatment), simply organize your data in a 2x2 table. Calculate the average grade in both groups before-and-after your exposure of interest. You can then perform the basic difference-in-differences calculation by hand. But again, I wouldn't recommend the extra effort. If you have many $i$ students and the data is stacked, simply estimate the foregoing regression. It is easier and you also obtain your standard errors—for free.

In the psychological sciences, it is common to observe applied researchers use the change score approach:

$$ Y_2 - Y_1 = \gamma + \beta T_i + \epsilon_i, $$

where the differences between two grades (i.e., $t_2 - t_1$) is regressed on a treatment dummy. This is yet another approach where you can keep your data in your desired structure (i.e., one individual $i$ per row). Your estimate of $\beta$ in the foregoing example should equal the coefficient on your interaction term in the stacked difference-in-differences regression. For more information on this approach, peruse this paper by Lüdtke and Robitzsch (2020), or review this post which also references the paper.

To conclude, you correctly note that swapping two students' grades in your treatment group should not affect your estimate in the simple 2x2 case with four cell means. I hasten to add that you shouldn't use a regression model on your "grouped" data frame which only includes four observations. Even though software may spit out coefficients, you have no residual degrees of freedom! Thus, you have no hope of calculating statistics in this setting. As a final word, I encourage you to stack your observations and estimate the second model outlined above. Let software do the work for you! This structure is handy once you start adding more students—and more time periods.