10
$\begingroup$

This is a broad question, I know, but I feel like it's ok to ask because I searched everywhere and I couldn't find the answer.

So I was studying about synthetic control (here: https://matheusfacure.github.io/python-causality-handbook/15-Synthetic-Control.html), and I thought about doing for an experiment I was running, but I ran into a problem: what can you do when your synthetic control is not so good? How can I improve it?

Let's use the book in the link as an example. There was a policy change in California and they evaluate the impact of the policy change by using California as the test group and by using all other states combined as a synthetic control group. The author did that by running a linear regression using the other states as the features and Califonia as the target. He then gets the coefficients from the regression and use it to build his synthetic group. But what if this coefficients are not so adherent? What if when you apply it to get the synthetic group, this group is not so similar as the place you are evaluating the change?

As already noted in the comments, there is one more step I didn't mention, the interpolation of the coefficients. Anyway, the problem is the same: let's suppose that you've tried the regression of all states and California, tried the interpolation, but the error is still big? I mean, suppose the regression+interpolation generated coefficients that didn't fit the data so well, suppose you applied the coefficients interpolated to actual data in order to see how well these coefficients could predict California data, but the error from the prediction these coefficients made is too big. Is there anything you could do to improve it? What should you do if you tried to build your synthetic control group but it didn't get so good.

$\endgroup$
2
  • 1
    $\begingroup$ Typically, SCM imposes the constraint that the weights are in [0,1) and sum to 1, and sparsity (most of the weights are zero). That is, the weights are not just coefficients from a regression of CA's outcome on the other states' outcomes. There's a discussion of this in the Extrapolations section at your link and this works fairly well. That would probably be the first thing to try. But your question is hard to address since we don't know exactly what you are doing. $\endgroup$
    – dimitriy
    Commented Jun 15, 2021 at 0:31
  • $\begingroup$ @DimitriyV.Masterov you are right, interpolation is another step in the pursue of a synthetic control. Regarding the difficulty in answering the question, just think with the example of the book. What would you do if the coefficients (+ interpolation) wasn't good enough? What would the author do if he tried a regression using all states as the independent variables and California as the dependent, tried the interpolation, but the graphic applying the coefficients interpolated wasn't so adherent? $\endgroup$
    – dsbr__0
    Commented Jun 15, 2021 at 11:07

2 Answers 2

4
$\begingroup$

In addition to the other excellent answer (and yes, more data can really help, this could include different levels of aggregation, e.g. instead of state-level data, can you get county-level data?), there's also the option of shortening or weighting the time horizon for which you are matching the synthetic control. E.g. in the book example, they could have gone further back (assuming the data are available), but they clearly made a choice that e.g. matching the trajectory for data from before the second world war is irrelevant to the question at hand. You could also decide that on a sliding scale of importance matching the data 1987 is twice as important as matching the data for 1977 (which you can either achieve by weighting the loss functions or just scaling the values you are matching - e.g. scaling standardized values in 1977 by a factor of $\sqrt{0.5} \approx 0.707$). This would be arguing that a similar recent trajectory is just more important than what happened further in the past.

However, there's a situation where interpolated synthetic controls just cannot match the treated unit. That's when it's the most extreme of the available units. E.g. int he book example, if California had had by far the highest (or the lowest) per capita cigarette sales prior to the intervention, there would just no weighted average of the other states that can match its numbers. While there might be ways around it (e.g. if you go to the county level and there are some counties somewhere that can match the California counties), this would also have been a big red flag warning you that perhaps no other states really look like California and trying to compare what happens in California to what happens in these states is really problematic. I.e. it might be a warning that there might be no good answer. To quote John Tukey:"The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data."

$\endgroup$
3
$\begingroup$

You can sometimes improve fit by:

  • Getting more pre-treatment data when possible, though that runs the risk of going so far back that the structural relationship is too different from the period in the study, like when CA was Mexico. Also, does not always work.
  • Adding variables that are not just the lagged outcome in calculating the weights, like beer consumption and income per capita.
  • Removing units from the potential donor pool that are dissimilar to the treated unit:
    • Only use untreated units that do not adopt interventions similar to the one under investigation during the period of the study.
    • Do not suffer large idiosyncratic/local shocks to the outcome of interest during the study period.
    • Have characteristics similar to the characteristics of the affected unit.
  • Fit can sometimes be improved by using a transformation of the dependent variable (e.g., $\Delta Y_{it}$ or $\frac{\Delta Y_{it}}{Y_{it}}$, so you are matching the trend or growth rather than the levels. Normalizing by population can help if your treated unit is large relative to the untreated, like CA would be had they not done per capita.
  • If the bias in fit is constant over time, you can simply subtract it from the effect. However, I have never seen that happen in the wild. -- If the poor fit is mostly in the early pre-treatment period, that period can be excluded if motivated well.
  • If the poor fit is in the late pre-treatment period, that could indicate evidence of treatment anticipation. For example, if consumers knew the tax was coming soon, they could stock up and put the tobacco in a freezer. Then moving back the treatment start date to the tax increase announcement can help.

If the fit is still bad, abandon the project and move on with your life or try something else.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.