0
$\begingroup$

Let's say that I want to run a causal inference experiment, that is an experiment on historical data for an intervention that we were not able to perform a randomized controlled trial for. In the case of something like a difference-in-differences (DD), or even just a basic linear/logit regression, for the purpose of estimating the causal impact (marginal effects in this case) of some intervention, is there a rule of thumb for attempting to control for the length of time to use in the pre-intervention period? In the past, I've at least tried to at least compare full weeks, in order to incorporate any weekday impact.

$\endgroup$
0
$\begingroup$

In the case of something like a difference-in-differences (DD), or even just a basic linear/logit regression, ... is there a rule of thumb for attempting to control for the length of time to use in the pre-intervention period?

In a difference-in-differences (DD) setting, we often want serial observations of our units before the treatment/policy of interest. Three or more pre-treatment time periods is desirable. The purpose of maximizing the temporal dimension pre-shock is to demonstrate, visually, parallel group trends. In particular, the treatment group and the control group should be moving in tandem before the treatment/policy goes into effect.

In the past, I've at least tried to at least compare full weeks, in order to incorporate any weekday impact.

It depends.

You may be observing cyclical and/or idiosyncratic shocks in the raw data. If they influence both groups, then we often model the common shocks with period dummies (i.e., time effects). However, if you’re observing a divergence in trend emerging over a long time horizon, then we may want to augment the DD equation to account for this. Suppose the treatment group and the control group were slowly proceeding on different growth trajectories in the pre-policy epoch. In practice, we may want to model this by giving each unit its own linear, or even quadratic, time trend. Note, this approach is only useful when you amass sufficient pre-policy data. Three or more time periods isn't enough in my opinion. In fact, group-specific time trends often require far more than three pre-treatment periods; it should be long enough that we could reasonably extrapolate the group-specific trend into the post-treatment period.

Suppose a policy is enacted in a subset of U.S. states to curb traffic fatalities. To investigate whether the policy actually resulted in less fatal crashes, you acquire yearly fatality data in all states before and after the policy. Assume the policy was introduced in 2018 and you were lucky enough to get your hands on state-level vehicular fatalities from 2000–2020. This results in 18 pre-treatment time periods, which is more than enough to graphically inspect the group trends and also sufficient to allow each state to have its own unique time trend. Peruse the top answer here for more information on how to model this in practice.

Even with a surplus of pre-event data, including group-specific time trends is still very restrictive as a modeling strategy, so don't overdue it. In practice, I would juxtapose your DD estimates across the restricted and unrestricted models. If the causal estimand is insensitive to alternative specifications, then your results appear more credible. Alternatively, its quite possible the group-specific trends may completely absorb your treatment effect, which is unfortunate.

In sum, it is sensible to acquire three or more pre-treatment observations in a DD application. Any less would invite skepticism.

$\endgroup$
5
  • $\begingroup$ Thank you for the answer and especially for the example, they always help with grounding an explanation. What do you mean by "group-specific"? Also, would there be an instance in which you include too many periods in the pre-event data - for instance, in your example, lets say that whatever data you're using in the pre-event control doesn't trend the same as your treatment group until the most recent 3 periods (years). In this case, would you cap the pre event period to 3 years? $\endgroup$ Apr 7 '21 at 22:05
  • $\begingroup$ The term "group" is a bit of a misnomer. I mean multiplying each panel unit (i.e., individual, district, county, state, etc.) with a continuous time trend variable. You may also see this approach referred to as unit-specific time trends. $\endgroup$ Apr 7 '21 at 22:14
  • $\begingroup$ As far as winnowing down your pre-treatment epoch goes, it is suspicious to say the least. Why would the group trends only move in tandem for a few periods? The "why" is important. Maybe measurement error explains the divergence in more distant periods. $\endgroup$ Apr 7 '21 at 22:21
  • $\begingroup$ Got it, yes that makes sense. I was thinking of a situation in which perhaps the landscape of traffic laws looked different before a certain period (example: for instance lets say everything was based on state laws that changed frequently up until 2015 when federal laws became the norm, something along those lines). I do agree though, most of my efforts with using regression approaches for casual inference have boiled down to the specific scenario understanding. The tough part has always been being comfortable with having accounted for most confounding variables with no way to test reliability $\endgroup$ Apr 7 '21 at 22:27
  • $\begingroup$ It might not make sense to assess outcome variation going back beyond a certain time point. Suppose you were only interested in motorcycle fatalities in a handful of jurisdictions throughout the United States. Maybe very little ridership was observed in previous decades, or outlawed entirely in some rural areas. It depends heavily upon what policy (i.e., treatment) is under evaluation. But note this is a content area matter, not a statistical one. $\endgroup$ Apr 8 '21 at 2:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.