6
$\begingroup$

I'd like to assess the impact of an upcoming policy implementation, as measured by changes in questionnaire response to a Likert-scale question.

I understand I could use a difference-in-difference approach. However, in my situation there is no single obvious comparison, non-treated population. I think I'd like to use the "Synthetic Control Method for Comparative Case Studies" as described by Abadie et al and implemented as Synth in R.

  • Alberto Abadie, Alexis Diamond, Jens Hainmueller. Journal of the American Statistical Association. June 1, 2010, 105(490): 493-505. doi:10.1198/jasa.2009.ap08746. full text

As summarized in the R help for synth:

synth estimates the effect of an intervention of interest by comparing the evolution of an aggregate outcome for a unit affected by the intervention to the evolution of the same aggregate outcome for a synthetic control group.

synth constructs this synthetic control group by searching for a weighted combination of control units chosen to approximate the unit affected by the intervention in terms of the outcome predictors. The evolution of the outcome for the resulting synthetic control group is an estimate of the counterfactual of what would have been observed for the affected unit in the absence of the intervention. [..] the synth function routinely searches for the set of weights that generate the best fitting convex combination of the control units. In other words, the predictor weight matrix V is chosen among all positive definite diagonal matrices such that MSPE is minimized for the pre-intervention period.

See also the useful summary by Srikant Vadali in answers below.

Is this method appropriate for survey/sampled data? Is there anything I need to do differently, or just use my Likert-response mean as the dependent variable? Any suggestions about how I'd power such a beast?

Thank you!

$\endgroup$
  • $\begingroup$ I find the question interesting. However, it would help if you could briefly summarize the idea in the paper so that users need not download and read the paper to answer your question. I think it would help a lot if your question is self-contained to get quick and quality answers. $\endgroup$ – user28 Oct 6 '10 at 15:46
  • $\begingroup$ Good suggestion. I'm not sure I understand the method well enough yet to summarize how it works accurately... in the mean time, I've excerpted the brief summary from the R implementation. $\endgroup$ – Heather Piwowar Oct 6 '10 at 20:46
3
$\begingroup$

[Caveat: I have not read the paper so the below may be nonsense for all I know ...]

Based on the summary of the R package I would venture to guess that you could use the proposed methodology for the survey data provided the following conditions are met:

  1. You have survey data from control groups during pre-intervention periods. These control groups need not be identical to the treatment groups.

  2. The data you have is time series data.

Provided points 1 and 2 are met my best intuition as to how the method works is as follows:

  1. First, construct a 'hypothetical' (synthetic in their words) control group that behaves as identical as possible as the treatment group. The hypothetical group is constructed by taking a convex combination of the control group data you have.

    As an example, suppose that you want to measure student performance on math. Your control groups could be different sections whereas the treatment group is one specific section. You construct the hypothetical control group such that the weighted (with the weights summing to 1 and hence convex) average of the scores of the control group sections is as close as possible to the scores of the treatment group before the intervention (i.e., use MSPE which is Mean Squared Prediction Error).

  2. Second, extrapolate the hypothetical group's scores into the post-intervention period using the parameter estimates from step 1.

Since, the hypothetical group has been constructed to be identical to the treatment group pre-intervention, the post-intervention scores of the hypothetical group provides an appropriate counter-factual evidence to the treatment group's post-intervention scores to assess the effectiveness of the intervention.

$\endgroup$
  • 1
    $\begingroup$ Thanks Srikant, yes I think that summarizes the method well. The examples I've seen have applied the method to an aggregate dependent variables like GDP or cigarette sales per capita (estimated from tax revenues). Is there anything special I'd need to do when using a survey-based dependent variable instead? I guess add confidence bands to my time-series plots?? Then do some power estimates based on what divergence between treated and control groups I wouldn't want to miss by chance, then extrapolate/guesstimate how many control-groups responses I need to form useful synthetic control?? $\endgroup$ – Heather Piwowar Oct 7 '10 at 12:53
  • $\begingroup$ Are there other things I'd need to do to take into account the fact that my dependent variable is an estimate? $\endgroup$ – Heather Piwowar Oct 7 '10 at 12:55
  • $\begingroup$ @heather Why do you call your DV as an estimate? If you call it as an estimate because you are taking the mean/sum of your likert scales then I do not see how that matters conceptually. As far as power is concerned, that would be dependent on the model/estimation specifics and I am unable to offer any suggestions on that issue. Perhaps, you can just email the authors for some suggestions or do a simulation to see how much sample size you need for desired power? $\endgroup$ – user28 Oct 7 '10 at 15:11
  • $\begingroup$ Great, Srikant, thanks. The fact that you don't think it matters conceptually answers it for me! I wanted confirmation of that I guess. $\endgroup$ – Heather Piwowar Oct 7 '10 at 16:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.