Difference-in-difference model with mediators: Estimating the effect of different elements of a policy

Question

How do I conduct a mediation analysis in a difference-in-difference setting? For example, a city selects some neighborhoods for a new crime fighting strategy (the treatment $D$) that involves an increase in the number of police officers on the street (mechanism $M_1$), additional surveillance cameras (mechanism $M_2$), an increase in misdemeanor arrests ($M_3$) and potentially other unmeasured components (e.g. change in leadership etc).

To estimate the effect of $D$ on crime, I can use a simple difference in difference model.

$$ y_{jt} = \delta_j + \gamma_t + \phi D_{jt} + \epsilon_{jt} $$

$y$ is the crime rate by neighborhood $j$ and year $t$. $D_{jt}$ signifies the neighborhoods and years in which the new crime fighting strategy was in place. The relevant coefficient for the overall effect of the treatment is $\phi$.

Now my question: How do I determine which element of the policy was effective (increase in officers, additional cameras, increase in misdemeanor arrests)? I see two relevant questions here:

• Did $M_x$ mediate the relation between $D$ and $Y$?
• What is the effect of $M_x$ on $y$ and can I use the variation in $M_x$ created by the policy to estimate the effect of $M_x$?

Answer 1

You need to explicitly think about a causal model for $Y$ including the $M^x$. It seems you are assuming the effect of $D$ on $Y$ is a constant $\phi$, so I'll assume constant effects throughout.

You could postulate that

$$ Y_{jt} = \delta^1_j + \gamma^1_t + \phi^1 D_{jt} + M_{jt}\alpha + \epsilon^1_{jt} $$

where $M_{jt}$ is a vector of all $M^x$ and $\alpha$ is a vector of coefficients. Then, $\phi^1$ is what the mediation literature would call the controlled direct effect of D, fixing the mediators $M^x$. Since we are assuming constant effects, this then also equals the so-called natural direct effect.

Under these assumptions, if $\phi^1$ is zero, then any causal effect $D$ has on $Y$ must flow through the mediators. It also holds under these assumptions that

$$ ATE = \phi = \phi^1 + NIE$$

where NIE stands for natural indirect effect, so that by identifying $\phi$ and $\phi^1$, you get the NIE "for free".

You assume that $\phi$ is identified via DiD. However, that does not automatically identify $\phi^1$. You need to assume that $\epsilon^1_{jt}$ is mean independent from D. In this context, this means to assume no unobserved variables besides $D$ and the unit/time FE influencing both $M$ and $Y$.

If this seems plausible, $\phi^1$ can be estimated by a regression of $Y$ on unit and time FE and D and M. That means that the usual advice not to adjust for post-treatment variables becomes moot, because we need to net out the indirect effect of $D$ through $M$.

Things get more a bit more tricky when the $M^x$ influence each other. However, this is testable under the no-confounding assumption: Simply regress any $M^x$ on $D$ and the other two mediators (and fixed effects). If any regression coefficient of the $M$s is different from zero, then either the no-confounding or the mediator-do-not-influence-each-other assumption is wrong.

If you're interested in the effect of $M^x$ on $Y$ only, under no-confounding, a simple FE regression of $Y$ on $M^x$ and $D$ will do under constant effects. Again, you then would also need to think about the relationship between the mediators in order to decide for which you needed to adjust in that regression.

Answer 2

For the second question, I think you can use $D$ as an instrumental variable for your $M_{x}$ variables (one $x$ at a time) like this. \begin{align} M_{jt}^{x} &= \lambda_j + \theta_t + \pi D_{jt} + \nu_{jt} \tag{First stage}\\ y_{jt} &= \delta_j + \gamma_t + \phi M_{jt}^{x} + \epsilon_{jt} \tag{Outcome equation} \end{align}

The interpretation of $\phi$ (using $M_{1}$ as an example) is the effect of an increase in the number of police officers on the street on neighborhood crime rate for those neighborhoods that have more police officers on the street because they were selected for a new crime fighting strategy (local average treatment effect).

The paper I have in mind is Stephens and Yang (2014), which is about education and schooling (and actually criticizing a commonly used approach in that literature), but anyway, that's where I got the idea from.