Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
Difference in differences has long been popular as a non-experimental tool, especially in economics. Can somebody please provide a clear and non-technical answer to the following questions about difference-in-differences.
What is a difference-in-difference estimator?
Why is a difference-in-difference estimator any use?
Can we actually trust difference-in-difference estimates?
What is a difference in differences estimator
Difference in differences (DiD) is a tool to estimate treatment effects comparing the pre- and post-treatment differences in the outcome of a treatment and a control group. In general, we are interested in estimating the effect of a treatment $D_i$ (e.g. union status, medication, etc.) on an outcome $Y_i$ (e.g. wages, health, etc.) as in
$$Y_{it} = \alpha_i + \lambda_t + \rho D_{it} + X'_{it}\beta + \epsilon_{it}$$
where $\alpha_i$ are individual fixed effects (characteristics of individuals that do not change over time), $\lambda_t$ are time fixed effects, $X_{it}$ are time-varying covariates like individuals' age, and $\epsilon_{it}$ is an error term. Individuals and time are indexed by $i$ and $t$, respectively. If there is a correlation between the fixed effects and $D_{it}$ then estimating this regression via OLS will be biased given that the fixed effects are not controlled for. This is the typical omitted variable bias.
To see the effect of a treatment we would like to know the difference between a person in a world in which she received the treatment and one in which she does not. Of course, only one of these is ever observable in practice. Therefore we look for people with the same pre-treatment trends in the outcome. Suppose we have two periods $t = 1, 2$ and two groups $s = A,B$. Then, under the assumption that the trends in the treatment and control groups would have continued the same way as before in the absence of treatment, we can estimate the treatment effect as $$\rho = (E[Y_{ist}|s=A,t=2] - E[Y_{ist}|s=A,t=1]) - (E[Y_{ist}|s=B,t=2] - E[Y_{ist}|s=B,t=1])$$
Graphically this would look something like this:
You can simply calculate these means by hand, i.e. obtain the mean outcome of group $A$ in both periods and take their difference. Then obtain the mean outcome of group $B$ in both periods and take their difference. Then take the difference in the differences and that's the treatment effect. However, it is more convenient to do this in a regression framework because this allows you
To do this, you can follow either of two equivalent strategies. Generate a control group dummy $\text{treat}_i$ which is equal to 1 if a person is in group $A$ and 0 otherwise, generate a time dummy $\text{time}_t$ which is equal to 1 if $t=2$ and 0 otherwise, and then regress $$Y_{it} = \beta_1 + \beta_2 (\text{treat}_i) + \beta_3 (\text{time}_t) + \rho (\text{treat}_i \cdot \text{time}_t) + \epsilon_{it}$$
Or you simply generate a dummy $T_{it}$ which equals one if a person is in the treatment group AND the time period is the post-treatment period and is zero otherwise. Then you would regress $$Y_{it} = \beta_1 \gamma_s + \beta_2 \lambda_t + \rho T_{it} + \epsilon_{it}$$
where $\gamma_s$ is again a dummy for the control group and $\lambda_t$ are time dummies. The two regressions give you the same results for two periods and two groups. The second equation is more general though as it easily extends to multiple groups and time periods. In either case, this is how you can estimate the difference in differences parameter in a way such that you can include control variables (I left those out from the above equations to not clutter them up but you can simply include them) and obtain standard errors for inference.
Why is the difference in differences estimator useful?
As stated before, DiD is a method to estimate treatment effects with non-experimental data. That's the most useful feature. DiD is also a version of fixed effects estimation. Whereas the fixed effects model assumes $E(Y_{0it}|i,t) = \alpha_i + \lambda_t$, DiD makes a similar assumption but at the group level, $E(Y_{0it}|s,t) = \gamma_s + \lambda_t$. So the expected value of the outcome here is the sum of a group and a time effect. So what's the difference? For DiD you don't necessarily need panel data as long as your repeated cross sections are drawn from the same aggregate unit $s$. This makes DiD applicable to a wider array of data than the standard fixed effects models that require panel data.
Can we trust difference in differences?
The most important assumption in DiD is the parallel trends assumption (see the figure above). Never trust a study that does not graphically show these trends! Papers in the 1990s might have gotten away with this but nowadays our understanding of DiD is much better. If there is no convincing graph that shows the parallel trends in the pre-treatment outcomes for the treatment and control groups, be cautious. If the parallel trends assumption holds and we can credibly rule out any other time-variant changes that may confound the treatment, then DiD is a trustworthy method.
Another word of caution should be applied when it comes to the treatment of standard errors. With many years of data you need to adjust the standard errors for autocorrelation. In the past, this has been neglected but since Bertrand et al. (2004) "How Much Should We Trust Differences-In-Differences Estimates?" we know that this is an issue. In the paper they provide several remedies for dealing with autocorrelation. The easiest is to cluster on the individual panel identifier which allows for arbitrary correlation of the residuals among individual time series. This corrects for both autocorrelation and heteroscedasticity.
For further references see these lecture notes by Waldinger and Pischke.
Wikipedia has a decent entry on this subject, but why not just use linear regression allowing for interactions between your independent variables of interest? This seems more interpretable to me. Then you might read up on analysis of simple slopes (in the Cohen et al book free on Google Books) if your variables of interest are quantitative.
It is a technique widely used in econometrics to examine the influence of any exogenous event in a time series. You pick two separate groups of data relating to before and after the event studied. A good reference to learn more is the book Introduction to Econometrics by Wooldridge.
Careful:
Two additional points are worth noting. First, 80 of the original 92 DD papers have a potential problem with grouped error terms as the unit of observation is more detailed than the level of variation (a point discussed by Donald and Lang [2001]). Only 36 of these papers address this problem, either by clustering standard errors or by aggregating the data. Second, several techniques are used (more or less informally) for dealing with the possible endogeneity of the intervention variable. For example, three papers include a lagged dependent variable in equation (1), seven include a time trend specific to the treated states, fifteen plot some graphs to examine the dynamics of the treatment effect, three examine whether there is an "effect" before the law, two test whether the effect is persistent, and eleven formally attempt to do triple-differences (DDD) by finding another control group. In Bertrand, Duflo, and Mullainathan [2002] we show that most of these techniques do not alleviate the serial correlation issues.
(Bertrand, Duflo, and Mullainathan 2004, 253)
