Synthetic Control Method difference between MSPE and RMSPE

Question

I am using Synthetic Control Method to estimate the effect some policy had. Got results and everything but when started to discuss inference method and placebo tests I have hit the wall. I got the $\frac{\mathrm{Post \; MSPE}}{\mathrm{Pre \; MSPE}}$ ratio from Synth() package, which I believed to be equal to p-value (at least that's what R reported). Yet, on numerous pages across the Internet I have found the information that the p-value is in fact equal to $\frac{\mathrm{Post \; RMSPE}}{\mathrm{Pre \; RMSPE}}$. Pretty sure, Synth() does not produce the aforementioned ratio - but then again, which one should I use?

Abadie et al. (2010) talks about MSPE - then 5 years later, Abadie et al. (2015) starts using RMSPE. I find it a little bit confusing and would really appreciate if someone could explain it to me.

Answer 1

The p-value comes from comparing how unusual your observed effect is relative to the false placebo effects (the ones where you run synth on untreated units). If it is unusual, you can reject the null of no effect.

You can use any measure of effect size that is comparable across units, but Abadie et al. (2010) suggest using post-treatment RMSPE (root mean square prediction error). That is the numerator of the ratio. The idea is that if there is an effect, the RMSPE will be large. So will be the MSPE. So if MSPE is 100, RMSPE is 10. If MSPE is 25, then RSMPE is 5. It does not really matter which you use.

So where does the denominator come from? The reason for the normalization by the pre RMSPE is that the placebo effects may be quite large if those units where not matched well in the pre-treatment period. This would cause p-values to be too conservative (i.e., you would not reject often enough). Adjusting for the quality of the pre-treatment matches can be done in two ways:

1. Restricting the comparison set of placebos to only include those that match well, throwing away FPs with pre RMSPE X times greater than treated. Abadie et al. use X=5 in the Basque terrorism example.
2. Dividing all post RMSPE by the corresponding pre-treatment match quality (RSMPE) to standardize them.

The second is simpler since it does not entail picking X in an ad hoc manner.

In the end, the two-sided p-value is the proportion of false placebo ratios that are greater or equal to the treated ratio. Some people will do that calculation with the high pre RMSPE placebos also thrown out. You can convince yourself of this by running a FP loop and storing the ratios along the way and doing the proportion calculation at the end.