2
$\begingroup$

Say you have a control group and an experimental group and you have verified the parallel trends assumption for them. Now say that at the time of intervention, the control group gets 9 separate interventions and the experimental group gets 10 interventions. If after the interventions you see a statistically significant difference would it be safe to say that it was due to the intervention that was in the experimental group but not in the control group? Or are the results invalid because one of the 9 interventions may have broken the parallel trends assumption by the time the 10th intervention was done?

$\endgroup$
4
  • $\begingroup$ Is treatment switching ‘on’ and ‘off’ for different units over a long time series? What do you mean when you say the control group received 9 separate treatments? Do you have a group of unexposed units? That is, is there a subset of units that never receive any intervention over your observation period? $\endgroup$ Nov 19, 2020 at 20:08
  • $\begingroup$ @ThomasBilach there would be no unexposed units. Both the control group and the experiment group would receive interventions. The only difference would be that the experiment group would receive one more additional intervention compared to the control group. $\endgroup$ Nov 19, 2020 at 21:15
  • $\begingroup$ And are these separate treatments? Are they qualitatively different? Are they implemented while the other one is still in effect? $\endgroup$ Nov 19, 2020 at 21:55
  • $\begingroup$ @ThomasBilach good question, in my case I cannot say exactly the order in which they take effect. Yes they are separate and qualitatively different. $\endgroup$ Nov 19, 2020 at 22:37

1 Answer 1

0
$\begingroup$

It will be difficult to disentangle the effects of each qualitatively distinct treatment if you do not know their order of implementation. It appears you know the onset of the initial treatment (otherwise you couldn't possibly make statements regarding trend equivalence), but not any of the subsequent interventions. If each exposure period is intermittent (i.e., 'on' and 'off' exposures experienced by a subset of units) then you could assess the stability of the group trends before each exposure period. But as you indicated in the comments, the onset of each intervention goes into effect while the previous one is in place.

Here are the potential problems as I see it. First, difference-in-differences requires you to observe a subset of unexposed units over time (i.e., pre- and post-treatment). In your case, you do not observe a group of nonadopter units. Your "never-receivers" represent your baseline history of never receiving treatment. Without a counterfactual, you cannot adequately estimate what would have happened to treated units/entities in the absence of treatment exposure. That being said, if the initial treatment is rolled out at different times for different units, then your estimate of a treatment effect is restricted to variation in treatment timing. But I don't think you can do this either. It appears the first onset of treatment affects all units at the same time. This brings me to my next concern.

Your second issue is you cannot adequately defend the validity of the method for any intervention beyond the initial treatment. The stability of your trends in any subsequent period might be completely offset by the institution of additional interventions, each of which might vary considerably in their duration and intensity. It is also unclear how much overlap there is between successive treatments, or if each is independent of the next. The difference between your groups is in their dosage of qualitatively distinct treatments.

If after the interventions you see a statistically significant difference would it be safe to say that it was due to the intervention that was in the experimental group but not the control group?

It's hard to say. It will be exceedingly difficult to disentangle the effect of each distinct treatment and how they differed across groups. I should also note that you've said nothing about the mechanism by which each intervention affects your outcome. All we know is one group received more exposures than the other.

Or are the results invalid because one of the 9 interventions may have broken the parallel trends assumption by the time the 10th intervention was done?

Your question implies that the length of each post-treatment epoch is known. If this is so, then I assume you want to use the periods after the ninth intervention in your control group as a counterfactual for the tenth intervention which is only in effect in the treatment group. This is problematic as the exposed group's pre-treatment series before the tenth intervention is also its post-period for the previous iteration. Estimates of a treatment effect may be biased if the effects from previous interventions change over time. In other words, a prior intervention might set one group on a completely different growth trajectory before the next exposure.

At a basic level, difference-in-differences requires you to observe a subset of untreated units/entities before and after your exposure(s) of interest. Without further information about the timing of each intervention and the intensity of exposure, it will be difficult to argue that any observed difference in trend is the result of the imposition of one additional shock in your treatment group.

I would also be hesitant labeling any group your "experimental" group, as each group is "treated" in their post-period.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.