0
$\begingroup$

I am in the process of setting up a difference-in-differences analysis (DiD) to evaluate a particular policy. It was launched on the year 2000, and it basically is relevant in ten-year periods. One important thing to notice is that different units (communities) started treatment in different years. I have full treatment information (2000 to 2020) and I have access to my outcome variable for the years 2000 to 2012 (it's GIS forest cover data). So that means, for the first 10 years of the policy, and two periods after it. I have a few questions here.

Does this mean I should mostly focus on the communities that were treated in 2000, 2001, and 2002 (given my outcome variable)?

Now, I have to do a pre-trend analysis too. I do have access to some data from previous years, but the pre-period data is:

  • a) of a lesser quality
  • b) non-continuous (so 1977 and 1984)
  • c) maybe doesn't cover all treated / non-treated units.

My first question is, would 2 time periods suffice? I believe I need 3 at least to test pre-trends (there is a good chance I do get 3 time periods, but want to know for sure).

My second question is, does the pre-policy time periods used for the pre-trend analysis have to be continuous (e.g., 1986 to 1991), or do they have to be adjacent to the date the policy was launched (e.g., 1995 until 1999)?

And third, does the pre-policy data HAVE to be of the same quality of the policy data (in GIS terms it's somewhat hard to find data for those years for the area I am looking into).

$\endgroup$
8
  • $\begingroup$ Welcome. When you say the data is “non-continuous” do you mean you have gaps in the times series in the pre-period? Or are you referring to the GIS forest cover data itself? $\endgroup$ Jan 29, 2021 at 16:52
  • $\begingroup$ Hello, and thank you for the comment. I am referring to the first, as of now, I have data for two years (for the pre-period), which have a gap (year 1984 and 1977) $\endgroup$
    – M.T
    Jan 29, 2021 at 19:29
  • $\begingroup$ What do you mean by the treatment is relevant in ten-year periods? And you also state that you have "full treatment information (2000 - 2020)" but then follow by saying that access to your outcome variable is from 2000 - 2012. This is a bit confusing. Could you elaborate further? $\endgroup$ Jan 29, 2021 at 19:39
  • $\begingroup$ or this policy, communties would receive property rights over a geographical area for 10 years. I have full information on all treated communities. My outcome variable is the observed deforestation on those land areas. I only have that data from 2000 to 2012 $\endgroup$
    – M.T
    Jan 29, 2021 at 20:12
  • $\begingroup$ So your outcome data does not span the actual full policy period. Then 2011 and 2012 serve as the two posttreatment (i.e., after) periods for some communities. Correct? And, due to the short treatment exposure period, you want to focus on communities treated earlier in your panel? $\endgroup$ Jan 29, 2021 at 20:13

1 Answer 1

0
$\begingroup$

Many of your questions cannot be answered definitively and require you to make some ad hoc decisions prior to estimating your model. I will address each question and comment, in turn.

I am in the process of setting up a difference-in-differences analysis (DiD) to evaluate a particular policy. It was launched on the year 2000, and it basically is relevant in ten-year periods. One important thing to notice is that different units (communities) started treatment in different years.

This is not a problem.

However, you must use the generalized difference-in-differences (DiD) estimator. This will account for the different subsets of communities with different policy adoption periods.

I have full treatment information (2000 to 2020) and I have access to my outcome variable for the years 2000 to 2012 (it's GIS forest cover data). So that means, for the first 10 years of the policy, and two periods after it.

It appears the time variation in your outcome does not span the full policy period. I would assume the late adopter communities have a longer treatment phase than the early adopters, but it isn't clear from your question. It isn't fatal to subject treated communities to a scanty number of post-treatment years. However, it will be difficult to nail down your treatment effect if it takes more than two years before the policy reaches its full effect. Or, suppose theory suggests effects accumulate over time. You cannot possibly address any long-run effects with a transient exposure epoch.

Does this mean I should mostly focus on the communities that were treated in 2000, 2001, and 2002 (given my outcome variable)?

Maybe.

If you only observe outcomes during the interval from 2000–2012, then I assume the early adopter communities (i.e., pre-2000) do not have any pre-treatment data prior to the year 2000. Thus, it might be worthwhile in your setting to restrict attention to the communities with policy adoption periods within the three-year window you referenced.

Your comments do suggest a subset of treated and/or untreated communities do not have any pre-policy data. But which ones? And how many? I just hope you don't lose too many clusters by dropping too many early adopter communities.

My first question is, would 2 time periods suffice? I believe I need 3 at least to test pre-trends (there is a good chance I do get 3 time periods, but want to know for sure).

Serial observations before policy adoption is not necessary—but is very much preferred. At least three pre-policy periods is recommended. It is rare today to find a DiD evaluation with less than three pre-treatment time periods.

My second question is, does the pre-policy time periods used for the pre-trend analysis have to be continuous (e.g., 1986 to 1991), or do they have to be adjacent to the date the policy was launched (e.g., 1995 until 1999)?

Your first parenthetical time interval suggests an 8-year gap before the intervention, assuming the year 2000 is the initial adoption year. Think of all the possible pseudo-paths your outcome could have taken pre-shock. For instance, suppose outcomes in treated communities trend upward before policy implementation, while untreated communities remain unchanged. Effects will be overstated in the post-period should outcomes in treated communities mean revert to pre-policy levels. How would you statistically model the pre-period spike with missing pre-policy data? You can't. In fact, you would have to make some strong theoretical assumptions about how your outcome was trending before the policy went into effect. Maybe the individuals nested within your communities anticipated the impending policy, which is often the case in settings where a sufficient time gap exists between the announcement of the policy and its ultimate effective date. Assessing anticipatory behavior may be worth exploring in periods approaching program exposure. But you have little hope of investigating any anticipatory effects if the years close to your initial treatment are entirely absent. In general, the years relative to treatment should be evenly spaced, and temporally adjacent to the policy adoption period.

Your second parenthetical time interval is preferred, which again is assuming the initial treatment is implemented in the year 2000. Each unit is observed over five evenly spaced yearly time periods pre-shock. Your years should move in regular intervals until treatment begins. Any gap in your time series invites deception into your plot(s) of the group trends, especially if you assume equidistant spacing over a multi-year gap. As a graphical recommendation: do not arbitrarily impose uniform intervals over an irregular time series. You do not want to be accused of visually manufacturing outcome variation over time. To borrow a line from Edward R. Tufte's book The Visual Display of Quantitative Information: "A steady canvas makes for a clearer picture" (p. 61). In sum, eschew any deception by arbitrary design variation. Instead, be transparent about what's missing.

If your time series is highly irregular in the pre-policy epoch, then you may want to consider using some interpolation methods to help transform the series into evenly spaced time units. However, this introduces bias if the missing time series information is long. A larger and larger void between the last pre-treatment period and the immediate (first) treatment year exacerbates the interpolation bias. Interpolate with caution.

And third, does the pre-policy data HAVE to be of the same quality of the policy data (in GIS terms it's somewhat hard to find data for those years for the area I am looking into).

It should.

This question suggests non-random measurement error in your outcome variable. Why is the resolution of lesser quality in the pre-period? Is it a software issue? If it's due to the absence of sufficient data pre-policy, then this is unfortunate. My only recommendation is to get more data. Should you select more distant pre-policy years further from the initial treatment, then I suppose this is better than no data at all. Just be prepared to defend your decisions!

$\endgroup$
7
  • $\begingroup$ Thank you very much for the thoughtful comment! and indeed getting more data is my current focus, and my questions were geared towards understanding what would be the preferred data to obtain. There is the opportunity that I will get outcome data since policy adoption (2000) until today, which would be good to keep my sample in a healthy size. Likewise, I also believe I will be able to get access to pre-policy deforestation data, hopefully 1995 to 1998, but the quality might be different (this is because of the resolution of GIS available, but I might be wrong here). $\endgroup$
    – M.T
    Jan 31, 2021 at 21:05
  • $\begingroup$ And as a follow-up, does the pre-policy data have to include all treated communities? There is a chance I have pre-policy outcome data for some regions more than others. $\endgroup$
    – M.T
    Jan 31, 2021 at 21:07
  • $\begingroup$ Is this a dichotomous treatment? And if so, that would mean a subset of treated communities in your panel start as treated entities and stay treated. How many of the approximately 60 treated communities don’t have any pre-policy data? $\endgroup$ Jan 31, 2021 at 22:26
  • $\begingroup$ Indeed, it basically can be interpreted as " given property rights" and not given. My treatment (as in the database) will be a dichotomous 1/0 variable.... once given treatment communities stay treated for 10 years (at least, since some renew immeadetely so they stay treated for 20 years). Its hard to say at the moment how many communities don't have pre-policy data.. I know that there is data on some areas more than others, but that is yet to be fully uncovered. Also 60 are currently treated but, its 94 communities which make up the sample (communities originally treated but currently not) $\endgroup$
    – M.T
    Feb 1, 2021 at 10:54
  • $\begingroup$ The unbalanced nature of your panel is not so much of a concern. Your biggest hurdle to overcome is identifying the communities without any pre-policy data. If a large subset of communities were not observed pre-shock, then you cannot possibly defend the validity of the method. Peruse my answer here for more information. I would subject the "always treated" entities to a separate analysis. $\endgroup$ Feb 1, 2021 at 22:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.