Odd behaviour in CausalImpact (R)

Question

I'm finding some odd behaviour in Google's CausalImpact R package and wondered if anyone has found the same and knows the cause. If you feed the package a certain length time series, the model snaps to a perfect historical fit, no matter what explanatory variables you use.

Using code from Google's own toy example, I set up a model, which works fine

library(CausalImpact)

total.points <- 300
marketing.starts <- 270

set.seed(1)
x1 <- 100 + arima.sim(model = list(ar = 0.999), n = total.points)
y <- 1.2 * x1 + rnorm(total.points)

y[marketing.starts:total.points] <- y[marketing.starts:total.points] + 10
data <- cbind(y, x1)

pre.period <- c(1, marketing.starts-1)
post.period <- c(marketing.starts, total.points)

impact <- CausalImpact(data, pre.period, post.period)

plot(impact)

This site won't let me post more than two image links as I'm new, but the above produces a regular CausalImpact example.

Now switch the explanatory variable X1 for a nonsense variable X2 (different seed) that doesn't explain y at all. The result is as you'd expect and the model no longer fits.

set.seed(10)
x2 <- 100 + arima.sim(model = list(ar = 0.999), n = total.points)

data <- cbind(y, x2)

impact <- CausalImpact(data, pre.period, post.period)

plot(impact)

Finally, change the historic and predicted periods, so that there is a bit more history and a shorter prediction. Still using only the nonsense X2 variable as explanatory.

total.points <- 300
marketing.starts <- 289

set.seed(1)
x1 <- 100 + arima.sim(model = list(ar = 0.999), n = total.points)
y <- 1.2 * x1 + rnorm(total.points)

y[marketing.starts:total.points] <- y[marketing.starts:total.points] + 10
data <- cbind(y, x2)

pre.period <- c(1, marketing.starts-1)
post.period <- c(marketing.starts, total.points)

impact <- CausalImpact(data, pre.period, post.period)

plot(impact)

The model suddenly has an almost perfect historical fit, even though I haven't given it anything useful to explain the past. It does it suddenly - if you use observation 288 as marketing start in the example above, it won't do it. I'm a newbie to the site, but would really appreciate any clues about what it's doing!

Answer 1

The behaviour is indeed simpler to reproduce with more data points. Reproducing an example similar to that in the follow-up comment to my first reply:

library(CausalImpact)

total.points <- 500
marketing.starts <- 295

set.seed(1)
x1 <- 100 + arima.sim(model = list(ar = 0.999), n = total.points)
y <- 1.2 * x1 + rnorm(total.points)
y[marketing.starts:total.points] <- y[marketing.starts:total.points] + 10

set.seed(10)
x2 <- 100 + arima.sim(model = list(ar = 0.999), n = total.points)
data <- cbind(y, x2)

pre.period <- c(1, marketing.starts-1)
post.period <- c(marketing.starts, total.points)
impact <- CausalImpact(data, pre.period, post.period)

plot(impact)

With the default tuning parameters (argument model.args), the CausalImpact fit mainly explains the time series with a strong random walk component and little noise superimposed. The prediction bands displayed in the post-intervention period reflect this fit by being narrow in the beginning and becoming broader later on. Fit and prediction bands are "valid", but probably not the best model fit given the knowledge about the time series.

One can force CausalImpact to attribute the fluctuations to noise rather than signal by adjusting the prior for the random walk component:

impact <- CausalImpact(data, pre.period, post.period,
                       model.args = list(prior.level.sd = 0.001))
plot(impact)

Note that the two plots look very similar when the models are fitted without any regressors at all. This shows that CausalImpact does not wrongly learn from the independent time series x2: the factor accounting for the tight fits is the random walk component, not the regressor. When using the useful regressor x1 instead of x2, the fit is also tight in the pre-intervention period, but it does not get as wide in the post-intervention period.

Answer 2

I can't reproduce your example... When I run the code provided, I get the following plot for the third case which looks OK:

So it seems to me that there is some different problem?