I am creating a synthetic control variable in which I want to use in order to perform a causal impact analysis of our marketing campaign. The goal is to find a set of predictor time series that are highly correlated with the treated time series prior to the marketing campaign and once the campaign begins the stationary relations between control and treated should break because the synthetic control time series is not exposed to the marketing campaign. During the campaign treatment period, the difference between the levels of the series should be attributed to the campaign effects. However, if the relationship between the synthetic control and treated series is spurious prior to the beginning of the marketing campaign, we can't attributed any differences between control and treated to the campaign effect. So I have heard that back testing the results is the way to validate whether the the relationships between the controls and treated time series is spurious. I cannot find "how to back test" steps or examples on the internet. I do see the same term used in relation to finance analysis, but if it is the same methodology, I don't know how one could apply it in the context of treated versus control. By the way, I am conducting this analysis in R using CausalImpact package.
I think there are two types of "false placebo"/back tests that work well with SC methods.
- Pick a few target time series where there was no treatment, like sales in a different store/region where the was no promotion and you don't expect any interference from the treated units. Repeat the analysis as if the treatment happened during your campaign period. If you see a lot of significant effects, become very nervous.
- Use the target time series from your campaign, but focus on the pre-campaign period. Pretend there was a campaign during a random day/week/month/year and repeat the same analysis few times. If you consistently find an effect, become worried.