3
$\begingroup$

I'm finding some odd behaviour in Google's CausalImpact R package and wondered if anyone has found the same and knows the cause. If you feed the package a certain length time series, the model snaps to a perfect historical fit, no matter what explanatory variables you use.

Using code from Google's own toy example, I set up a model, which works fine

library(CausalImpact)

total.points <- 300
marketing.starts <- 270

set.seed(1)
x1 <- 100 + arima.sim(model = list(ar = 0.999), n = total.points)
y <- 1.2 * x1 + rnorm(total.points)

y[marketing.starts:total.points] <- y[marketing.starts:total.points] + 10
data <- cbind(y, x1)

pre.period <- c(1, marketing.starts-1)
post.period <- c(marketing.starts, total.points)

impact <- CausalImpact(data, pre.period, post.period)

plot(impact)

This site won't let me post more than two image links as I'm new, but the above produces a regular CausalImpact example.

Now switch the explanatory variable X1 for a nonsense variable X2 (different seed) that doesn't explain y at all. The result is as you'd expect and the model no longer fits.

set.seed(10)
x2 <- 100 + arima.sim(model = list(ar = 0.999), n = total.points)

data <- cbind(y, x2)

impact <- CausalImpact(data, pre.period, post.period)

plot(impact)

enter image description here

Finally, change the historic and predicted periods, so that there is a bit more history and a shorter prediction. Still using only the nonsense X2 variable as explanatory.

total.points <- 300
marketing.starts <- 289

set.seed(1)
x1 <- 100 + arima.sim(model = list(ar = 0.999), n = total.points)
y <- 1.2 * x1 + rnorm(total.points)

y[marketing.starts:total.points] <- y[marketing.starts:total.points] + 10
data <- cbind(y, x2)

pre.period <- c(1, marketing.starts-1)
post.period <- c(marketing.starts, total.points)

impact <- CausalImpact(data, pre.period, post.period)

plot(impact)

enter image description here

The model suddenly has an almost perfect historical fit, even though I haven't given it anything useful to explain the past. It does it suddenly - if you use observation 288 as marketing start in the example above, it won't do it. I'm a newbie to the site, but would really appreciate any clues about what it's doing!

$\endgroup$
6
$\begingroup$

The behaviour is indeed simpler to reproduce with more data points. Reproducing an example similar to that in the follow-up comment to my first reply:

library(CausalImpact)

total.points <- 500
marketing.starts <- 295

set.seed(1)
x1 <- 100 + arima.sim(model = list(ar = 0.999), n = total.points)
y <- 1.2 * x1 + rnorm(total.points)
y[marketing.starts:total.points] <- y[marketing.starts:total.points] + 10

set.seed(10)
x2 <- 100 + arima.sim(model = list(ar = 0.999), n = total.points)
data <- cbind(y, x2)

pre.period <- c(1, marketing.starts-1)
post.period <- c(marketing.starts, total.points)
impact <- CausalImpact(data, pre.period, post.period)

plot(impact)

tight fit with large prediction band

With the default tuning parameters (argument model.args), the CausalImpact fit mainly explains the time series with a strong random walk component and little noise superimposed. The prediction bands displayed in the post-intervention period reflect this fit by being narrow in the beginning and becoming broader later on. Fit and prediction bands are "valid", but probably not the best model fit given the knowledge about the time series.

One can force CausalImpact to attribute the fluctuations to noise rather than signal by adjusting the prior for the random walk component:

impact <- CausalImpact(data, pre.period, post.period,
                       model.args = list(prior.level.sd = 0.001))
plot(impact)

broad, but constant prediction bands

Note that the two plots look very similar when the models are fitted without any regressors at all. This shows that CausalImpact does not wrongly learn from the independent time series x2: the factor accounting for the tight fits is the random walk component, not the regressor. When using the useful regressor x1 instead of x2, the fit is also tight in the pre-intervention period, but it does not get as wide in the post-intervention period.

$\endgroup$
1
$\begingroup$

I can't reproduce your example... When I run the code provided, I get the following plot for the third case which looks OK:

enter image description here

So it seems to me that there is some different problem?

$\endgroup$
  • $\begingroup$ That's odd, I've got two different PC's doing it and just copied and pasted the code above to check it's exactly what causes the problem. I definitely get the last chart I included in my post. It does seem to be dependent on the seeds rather than always happening at a specific number of observations. More obs in the prior period makes it more likely to happen. It seems like at some point, causalimpact becomes sure the X variable isn't explanatory and switches over. This switches the obs to 500 & 450 and should run on its own without the code above (next comment) $\endgroup$ – NeilC Sep 12 '16 at 8:05
  • $\begingroup$ total.points <- 500 marketing.starts <- 450 set.seed(1) x1 <- 100 + arima.sim(model = list(ar = 0.999), n = total.points) y <- 1.2 * x1 + rnorm(total.points) y[marketing.starts:total.points] <- y[marketing.starts:total.points] + 10 set.seed(10) x2 <- 100 + arima.sim(model = list(ar = 0.999), n = total.points) data <- cbind(y, x2) pre.period <- c(1, marketing.starts-1) post.period <- c(marketing.starts, total.points) impact <- CausalImpact(data, pre.period, post.period) plot(impact) No line breaks? That's annoying. $\endgroup$ – NeilC Sep 12 '16 at 8:10
  • $\begingroup$ Sorry for the late follow-up! With your new example, I could reproduce the problem; for a solution, see my reply below. $\endgroup$ – Alain Hauser Oct 11 '16 at 13:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.