1
$\begingroup$

I am trying to get a good understanding of Difference-in-Difference (DiD) analysis. I will use the sample data constructed in R below.

set.seed(1)    
data=data.frame(State = c(rep("CA",8),rep("TX",8),rep("NY",8),rep("FL",8)),
                        Period = rep(c(1:8),4),
                        unemployment_rate= c(sample(1:10,8,replace=TRUE),
                                   sample(1:10,8,replace=TRUE),
                                   sample(100:150,8,replace=TRUE),
                                   sample(1000:1050,8,replace=TRUE)),
                        trt=c(rep(1,8),rep(0,24)))  
        
        data$time =factor(ifelse(data$Period <=4,"0_Pre","1_Post"))
        data$mult_period =as.factor(ifelse(data$Period <=4,0,data$Period-4))

Assume I want to evaluate the impact of a policy implemented in California on the unemployment rate by using DiD where Texas, New York, and Florida represent my control group.

Question 1: Would the correct way to formulate the model to get the policy impact be the following where the interaction term would be my impact?

did1 = lm(unemployment_rate ~ trt*time, data = data)
summary(did1 )
Coefficients:
                 Estimate Std. Error t value Pr(>|t|)   
(Intercept)     3.893e+02  1.219e+02   3.195  0.00345 **
trt            -3.841e+02  2.437e+02  -1.576  0.12627   
time1_Post      2.991e-14  1.723e+02   0.000  1.00000   
trt:time1_Post -1.750e+00  3.447e+02  -0.005  0.99598 

Question 2: This model does not take into account which state in the control group the observations came from or that multiple measurements where made in the same state over time. The model only considers if the time period was before or after the policy change and if they were treated, is this correct? If so are there any drawbacks and should repeated measures or mixed effects models be used instead?

Question 3: Does the large difference in scale between the states for unemployment_rate matter?

Question 4: If I fit the following model do the interaction terms represent the policy impact of each period after the policy change relative to the entire pre policy period?

did2 = lm(unemployment_rate  ~ trt*mult_period, data = data)
summary(did2 )
Coefficients:
                 Estimate Std. Error t value Pr(>|t|)   
(Intercept)       389.333    137.469   2.832   0.0097 **
trt              -384.083    274.937  -1.397   0.1764   
mult_period1       -8.000    307.389  -0.026   0.9795   
mult_period2        6.333    307.389   0.021   0.9837   
mult_period3       -2.000    307.389  -0.007   0.9949   
mult_period4        3.667    307.389   0.012   0.9906   
trt:mult_period1    4.750    614.779   0.008   0.9939   
trt:mult_period2   -4.583    614.779  -0.007   0.9941   
trt:mult_period3   -1.250    614.779  -0.002   0.9984   
trt:mult_period4   -5.917    614.779  -0.010   0.9924
$\endgroup$
2
  • $\begingroup$ Welcome. I just tried to reproduce your results but the coefficients don't match. Is it possible you ran the equation with a different seed number? $\endgroup$ Commented Mar 28, 2021 at 20:23
  • $\begingroup$ I get the same numbers not sure what happened I am interested in the method and interpretation more so any numbers would be fine to answer the questions. $\endgroup$ Commented Mar 29, 2021 at 2:28

1 Answer 1

1
$\begingroup$

I reproduced your dataset for the purposes of this answer. I sampled random deviates to make the output a little more digestible. Note the standard errors in your output shoot through the roof, in part because you're sampling integers from a very narrow range of values. In general, your code will return a difference-in-differences (DiD) estimate.

library(tidyr)

set.seed(123)

df <- tibble(
  state = c(rep("CA", 8), rep("TX", 8), rep("NY", 8), rep("FL", 8)),
  period = rep(c(1:8), 4),
  unemployment_rate = c(sample(rnorm(1000, 20, 2), 8, replace = TRUE), 
                        sample(rnorm(1000, 10, 1), 8, replace = TRUE),
                        sample(rnorm(1000, 10, 1), 8, replace = TRUE),
                        sample(rnorm(1000, 10, 1), 8, replace = TRUE)),
  trt = ifelse(state == "CA", 1, 0),  # equals 1 for California (only California is treated)
  post = ifelse(period > 4, 1, 0),    # equals in all periods after the policy in both groups
  mult_period = as.factor(ifelse(period <= 4, 0, period - 4))
  )

Question 1: Would the correct way to formulate the model to get the policy impact be the following where the interaction term would be my impact?

Yes.

The interaction term is the DiD estimate.

Question 2: This model does not take into account which state in the control group the observations came from or that multiple measurements where made in the same state over time. The model only considers if the time period was before or after the policy change and if they were treated, is this correct? If so are there any drawbacks and should repeated measures or mixed effects models be used instead?

In settings with two groups and two discrete time periods, the coefficient is easily computed using the "grouped" data. Since you're only working with country averages and not considering any other time-varying covariates, then we can work out the calculations by hand. This does not absolve you from demonstrating parallel group trends before the intervention. Don't just ignore this. The trends matter.

Question 3: Does the large difference in scale between the states for unemployment_rate matter?

No.

The level differences across countries is not what matters. What does matter is their trends across time. The mean unemployment rate in the treatment group and the mean unemployment rate in the control should be moving in tandem prior to policy adoption. To be clear, the group averages in each time period need not overlap. The mean unemployment rate for treated countries is allowed to be lower (or higher) than the mean unemployment rate for untreated countries. What we should expect, though, is a parallel evolution of the trends in each group. Bias is introduced if the two groups are proceeding on entirely different growth trajectories pre-shock.

I plotted the group trends below using my fake data. The red line denotes the trend in the treatment group (i.e., California); the blue line denotes the trend in the control group (i.e., Texas, Florida, and New York). To be clear, this is graphical evidence of a violation of the parallel trends assumption. California is trending upward pre-policy whereas the trend in the control group is relatively flat. As indicated earlier, the level differences do not matter, but how they trend over time does! Remember, the equation is computing a double difference across groups and across times, and so the stable, time-invariant differences across groups are essentially 'differenced out' using this approach.

library(ggplot2)
library(dplyr)

df %>%
  group_by(trt, period) %>%  # group by the treatment dummy and the time index
  summarize(avg = mean(unemployment_rate)) %>%
    ggplot(., aes(x = factor(period), y = avg, color = factor(trt, levels = c(1, 0)))) +
    geom_line(aes(group = trt)) +
    labs(x = "Time", y = "Mean Unemployment Rate") +
    geom_vline(xintercept = 5, color = "black", linetype = "dashed") +
    scale_color_manual(name = "Group:",
                       values = c("firebrick", "dodgerblue"),
                       labels = c("Treatment", "Control")) +
    theme_classic()

Violation of the Parallel Trends Assumption

It is clear based upon this plot of the group averages that some trend is already emerging in California before the policy.

Question 4: If I fit the following model do the interaction terms represent the policy impact of each period after the policy change relative to the entire pre policy period?

Yes.

This approach has its advantages. Effects may grow or fade over time. Note, R automatically interacts trt with each period after the policy. Periods 1–4 serve as the reference period. At first glance, it appears as if you interacted trt with pre-period dummies given your numeric labeling scheme, but this isn't the case. Technically, the appended labels correspond to periods 5–8. It doesn't affect estimation, but just be clear about the contrasts you're making in this setting.


Suggestions

I generally favor assigning names to variables that aid our memory. I created the variable post which is equal to 1 in periods 5–8, 0 otherwise; this is your post-treatment indicator. Interacting the two indicator variables will return the DiD estimate.

Here is the classical DiD approach. Note, R estimates the constituent terms for free:

> round(summary(lm(unemployment_rate ~ trt * post, data = df))$coefficients, 2)

            Estimate Std. Error t value Pr(>|t|)
(Intercept)     9.92       0.31   31.83     0.00
trt             8.14       0.62   13.06     0.00
post            0.37       0.44    0.83     0.41
trt:post        3.19       0.88    3.62     0.00

Now let's see how effects evolve over time post-shock:

> round(summary(lm(unemployment_rate ~ trt * mult_period, data = df))$coefficients, 2)

                 Estimate Std. Error t value Pr(>|t|)
(Intercept)          9.92       0.29   34.42     0.00
trt                  8.14       0.58   14.13     0.00
mult_period1         0.69       0.64    1.07     0.30
mult_period2         0.42       0.64    0.65     0.52
mult_period3        -0.19       0.64   -0.30     0.77
mult_period4         0.54       0.64    0.84     0.41
trt:mult_period1     3.84       1.29    2.98     0.01
trt:mult_period2     0.61       1.29    0.48     0.64
trt:mult_period3     3.90       1.29    3.03     0.01
trt:mult_period4     4.43       1.29    3.44     0.00

As a check on the validity of the DiD design, applied researchers often saturate the model with a full series of $T - 1$ time (period) dummies. In general, you shouldn't be observing strong non-zero effects in the periods prior to policy adoption. To achieve this, simply interact trt with a factorized version of period:

> round(summary(lm(unemployment_rate ~ trt * as.factor(period), data = df))$coefficients, 2)

                       Estimate Std. Error t value Pr(>|t|)
(Intercept)                9.56       0.58   16.38     0.00
trt                        9.69       1.17    8.30     0.00
as.factor(period)2        -0.02       0.83   -0.02     0.98
as.factor(period)3         0.80       0.83    0.97     0.35
as.factor(period)4         0.65       0.83    0.79     0.44
as.factor(period)5         1.05       0.83    1.27     0.22
as.factor(period)6         0.78       0.83    0.94     0.36
as.factor(period)7         0.17       0.83    0.20     0.84
as.factor(period)8         0.90       0.83    1.09     0.29
trt:as.factor(period)2    -2.44       1.65   -1.48     0.16
trt:as.factor(period)3    -2.63       1.65   -1.59     0.13
trt:as.factor(period)4    -1.14       1.65   -0.69     0.50
trt:as.factor(period)5     2.29       1.65    1.39     0.18
trt:as.factor(period)6    -0.94       1.65   -0.57     0.58
trt:as.factor(period)7     2.35       1.65    1.42     0.17
trt:as.factor(period)8     2.88       1.65    1.74     0.10

Note, we must leave out one time dummy. Software excludes the first period by default (i.e., period 1). I've reviewed papers where the authors dropped the period immediately before policy adoption (i.e., period 4). I don't think there's a consensus concerning which period to omit, though dropping more distant periods seems to be less common in practice. If you want to adjust the reference period programmatically, then simply re-estimate the model using this dataset:

df_relevel <- df %>% 
  mutate(period = relevel(factor(period), ref = ...))  # specify the period you want dropped

In a fully saturated model, the effects observed before and after the immediate adoption period are often referred to as 'leads' and 'lags' of treatment, respectively. If the policy takes a few periods before you observe any meaningful effect on unemployment, then we often say that the policy affects unemployment with a lag.

Try out these different specifications!

$\endgroup$
3
  • $\begingroup$ Great answer! Could you give a little more depth on question 2 specifically can I alter DiD to use the fact that observations are paired or alternative models which will allow it. $\endgroup$ Commented Mar 29, 2021 at 6:33
  • $\begingroup$ No problem. In your second question you ask whether you should use a “repeated measures” model. But what do you mean exactly? DiD methods require “repeated measures” to work. In your case, you observe the same country repeatedly over several time periods. $\endgroup$ Commented Mar 29, 2021 at 16:29
  • $\begingroup$ I was thinking along the lines of using a repeated measures design where you can model the covariance structure among measurements. Idk I will do further research your answer is great. Thanks $\endgroup$ Commented Mar 30, 2021 at 0:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.