Can I use Synthetic Control Method for Comparative Case Studies with survey data?

Question

I'd like to assess the impact of an upcoming policy implementation, as measured by changes in questionnaire response to a Likert-scale question.

I understand I could use a difference-in-difference approach. However, in my situation there is no single obvious comparison, non-treated population. I think I'd like to use the "Synthetic Control Method for Comparative Case Studies" as described by Abadie et al and implemented as Synth in R.

• Alberto Abadie, Alexis Diamond, Jens Hainmueller. Journal of the American Statistical Association. June 1, 2010, 105(490): 493-505. doi:10.1198/jasa.2009.ap08746. full text

As summarized in the R help for synth:

synth estimates the effect of an intervention of interest by comparing the evolution of an aggregate outcome for a unit affected by the intervention to the evolution of the same aggregate outcome for a synthetic control group.

synth constructs this synthetic control group by searching for a weighted combination of control units chosen to approximate the unit affected by the intervention in terms of the outcome predictors. The evolution of the outcome for the resulting synthetic control group is an estimate of the counterfactual of what would have been observed for the affected unit in the absence of the intervention. [..] the synth function routinely searches for the set of weights that generate the best fitting convex combination of the control units. In other words, the predictor weight matrix V is chosen among all positive definite diagonal matrices such that MSPE is minimized for the pre-intervention period.

See also the useful summary by Srikant Vadali in answers below.

Is this method appropriate for survey/sampled data? Is there anything I need to do differently, or just use my Likert-response mean as the dependent variable? Any suggestions about how I'd power such a beast?

Thank you!

Answer 1

[Caveat: I have not read the paper so the below may be nonsense for all I know ...]

Based on the summary of the R package I would venture to guess that you could use the proposed methodology for the survey data provided the following conditions are met:

1. You have survey data from control groups during pre-intervention periods. These control groups need not be identical to the treatment groups.

2. The data you have is time series data.

Provided points 1 and 2 are met my best intuition as to how the method works is as follows:

1. First, construct a 'hypothetical' (synthetic in their words) control group that behaves as identical as possible as the treatment group. The hypothetical group is constructed by taking a convex combination of the control group data you have.

   As an example, suppose that you want to measure student performance on math. Your control groups could be different sections whereas the treatment group is one specific section. You construct the hypothetical control group such that the weighted (with the weights summing to 1 and hence convex) average of the scores of the control group sections is as close as possible to the scores of the treatment group before the intervention (i.e., use MSPE which is Mean Squared Prediction Error).

2. Second, extrapolate the hypothetical group's scores into the post-intervention period using the parameter estimates from step 1.

Since, the hypothetical group has been constructed to be identical to the treatment group pre-intervention, the post-intervention scores of the hypothetical group provides an appropriate counter-factual evidence to the treatment group's post-intervention scores to assess the effectiveness of the intervention.