1
$\begingroup$

I have located a natural experiment in a time series cross-sectional dataset, but I am unsure of whether or not to include unit-level fixed effects in my models. I have produced a toy example that explains the situation:

Say I have an experiment in which I observe some outcome for a group of units over time. Now, some are randomly assigned to receive a treatment at a specific time $t^*$. For two units, the data might look like this:

unit time outcome treatment

1    1       2         0
1    2       2         0
1    3       2         0
1    4       2         0
1    5       2         0
2    1       2         0
2    2       2         0
2    3       6         1      <- t^*
2    4       6         0
2    5       2         0

Say I then attempt to estimate the effect of the treatment by regressing $outcome$ on $treatment$, like so:

$outcome_{it} = \beta_0 + \beta_1treatment_{it} + \epsilon_{it}$

With this sample, $\beta_1 = 3.6$. But now, having variation over time for each unit, I might try to include unit-level fixed effects. The demeaned data looks like this:

unit time outcome treatment outcome.demean treatment.demean

1    1       2         0         0               0
1    2       2         0         0               0
1    3       2         0         0               0
1    4       2         0         0               0
1    5       2         0         0               0 
2    1       2         0       -1.6             -0.2
2    2       2         0       -1.6             -0.2
2    3       6         1        2.4              0.8
2    4       6         0        2.4             -0.2 
2    5       2         0       -1.6             -0.2 

Estimating the fixed effects model...

$outcome.demean_{it} = \pi_0 + \pi_1treatment.demean_{it} + \mu_{it}$

...yields $\pi_1=3$. A smaller effect. This is clearly due to the fact that the effect of treatment seems to last two periods, the second of which is not marked by the treatment dummy, but is still included in the mean that is being subtracted from the outcome values of the treated unit. Thus I would argue that the fixed effects are "post-treatment" in this situation, because some of the values used in their calculation are consequences of the treatment without being incorporated into the treatment period. I would therefore opt not to include fixed effects here.

Am I correct in reasoning like this?

Note: Of course, the correct solution in this example would probably be to explicitly account for the lasting effect of the treatment by a lag or something equivalent, but in the actual setting where I encountered this problem, that is not feasible.

Note 2:* Also, given the as-if-randomness of assignment, fixed effects should not be needed to obtain an unbiased estimate of the ATE. However, some people may be less convinced by the natural experiment than I am and might therefore call for fixed effects to be included.

$\endgroup$
  • $\begingroup$ Cross-sectional is so named because it is a single point in time. I think you mean prospective, longitudinal, but not panel data. But can you describe the "units" are they students within schools, teeth within a mouth, etc? Please see stats.stackexchange.com/tags/analysis/info for tips on how to edit your question $\endgroup$ – AdamO Dec 15 '15 at 4:12
  • $\begingroup$ In social science it is fairly common to refer to data with multiple units observed at multiple time periods as "time series cross-sectional" (TSCS). Although some authors argue for a distinction, panel and TSCS are often used interchangeably. The units in my study are pairs of countries observed once every year, but the question really applies to any situation in which you have a lasting treatment effect and units observed over time. $\endgroup$ – Bertel Dec 15 '15 at 18:57
0
$\begingroup$

Your reasoning is not correct. First, if your experiment is randomized, treatment cannot be a (causal) function of any observables or fixed effects. So the fixed effects could indeed only be functions of the treatment, but I don't think that makes sense substantively. In any case, you would not need to adjust for the fixed effects, but you could benefit statistically from random effects.

Secondly, in your concrete example, the difference between the regression coefficients rests on the fact that they are also misspecified; by construction of the data, the treatment effect materializes over two periods.

$\endgroup$
  • $\begingroup$ Hi @Julian Schuessler, thank you for your answer! I have a few clarifying questions: 1) Why does it not make sense substantively that a treatment assigned at some point affects the outcome in more than just the exact period in which it was assigned? (if it does, the fixed effects would, in part, be a function of treatment). 2) Yes, that's the point! : ) When the effect materializes over time and lags are not specified correctly, then the fixed effects soak up the effect and introduce downwards bias in the coefficient on treatment; same as any other "post-treatment" variable. Right?. $\endgroup$ – Bertel Feb 1 '16 at 18:16
  • $\begingroup$ @Bertel: 1) That makes sense, but the fixed effects still cannot be a function of the treatment if it is randomized. Should treatment not be randomized, I do not immediately see how a fixed effect could be a causal function of the treatment, in a substantive sense 2) The fixed effects do not introduce bias, they might increase it. I do not see why this bias incrase should be in one or the other direction. When one (wrongly) conditions on post-treatment variables, the bias can go in any direction. $\endgroup$ – Julian Schuessler Feb 10 '16 at 19:34
  • $\begingroup$ Sorry for the slow reply (been travelling)! 1) I think there are some terminology-issues here. What I mean is that the value that is being subtracted from each observation (the unit-means = fixed effects) is affected by the treatment. Surely, this makes sense - the treatment had a positive effect not adequately captured by the dummy, so it increases the means being subtracted. 2) True, post-treatment bias can in general be in either direction, but in a case like above, it would seem to always be towards zero, since the means being subtracted are affected by the treatment. $\endgroup$ – Bertel Mar 10 '16 at 2:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.