How do I measure impact of an intervention of time series without historical data of the same series?

Question

I'm trying to evaluate the impact of an intervention in the most statistically accurate way possible but I'm having a hard time figuring out a way measure incremental growth along with statistical validation (for e.g., MAPE for forecasting). The problem with this data is that there is no historical data to forecast or run a causalImpact on. This is because the product marketing intervention and product launch happens simultaneously, leaving no room for a non-intervention period.

Here is the sample data which contains daily time series sales of 4 Cars. One is a test car which has been exposed to a marketing intervention. The other 3 are control cars which are similar to the test in terms of the category of the product, price and launch season of product. The scale for these control cars will be different due to factors such as brand size. Additionally, among the 3 control cars, Control car #2 has its launch in the same quarter but different year(2016) when compared to the test product(2017).

To measure incremental growth, this is what I've done so far ( Tab 'Normalization and Scaling' of the data attached) :

1. (Min-Max)Normalized all the control cars sales to a range 0 to 1.
2. Take the average of the 3 normalized control cars
3. Scale the averaged Normalizations to the ranges of the test group to seem like a similarly scaled baseline.
4. The difference between the Test car and the scaled data points should give me the daily incremental.

However, the massive problem with this method is that there is no way to measure accuracy to this method. Clearly, this is a flawed method. Is there an alternative method? How do I make this more statistically sound and logical?

Answer 1

Short answer: Your data likely do not contain the answer to your question and there is no general way to make up for it with statistics.

Long answer: As you said, any differences in sales can be easily attributed to inherent differences between the cars, not the intervention. There is no general way to decouple these inherent differences from the differences in marketing. Your data simply are not very informative (in particular, there is no "A/B test" - that would be if you had multiple sales points, only some of the running the intervention).

The only way out of this is to make strong assumptions about the way sales are determined. Your post implies you are willing to assume that the inherent differences have a linear effect on sales (they simply change the scale of the time series). If the intervention is supposed to change the shape of the time series (e.g. the intervention adds a peak at the beginning, but does not affect the tail), you might be able to detect it. A simple way to do this would be to run a linear regression to predict the test car sales for each day from the sales of the control cars. Then look at the residual error over time and see if there is some systematic trend.

A more thorough way might be to use Bayesian statistics and fit a spline or a Gaussian process to the time series (one curve for all the control cars together, one curve for the test car + a separate scale coefficient for each car) and then look at the distribution of the differences between the two curves. You might also need to treat weekends/holidays in a special way. This will likely require a custom model coded in Stan or similar language, although the brms package might have you covered.

But the assumptions made above are almost certain to be false: inherent differences may have (at least partially) non-linear effect and the intervention will have (at least partially) a linear effect. Also most models will assume Gaussian noise in the sales which might not be plausible. You need to be very careful and use your judgement to check if it all makes sense - there is IMHO no general way to even guess the size of possible error.

Combining sales from different times is also tricky, at the very least you need to take into account weekends and holidays.