CausalImpact analysis giving bad predictors

Question

I am quite new to causal impact. I have been using the python package 'causalimpact' since I read good feedback online and decided to give it a go.

For context I am trying to check the impact of a marketing campaign, that happened multiple times throughout a given day (starting at 5.44 PM), on the visits. I am using two other categories (that did not get any marketing) on the same website as control variables as seen bellow:

However, I have a big problem. The prediction seems to do an horrible job and I don't know why.

It seems that the prediction almost completely misses the massive decline of visits after 10 PM and is severely more optimistic.

For other days where the marketing campaign starts earlier it seems to do well later on the day but is less optimist in regards to visits for most of the day:

Anything I am doing wrong here?

Any help welcome!

Answer 1

I don't see anything wrong with what you have done. Maybe you are expecting too much from CausalImpact. Anyway, here are a couple of suggestions that you could try to improve your results:

Independence: For CausalImpact to work it is decisive that your control variables x1 and x2 are independent of your interventions, i.e., of the marketing campaign. This is probably very difficult to check. For example, if those controls are the visits for other products on the same site, depending on your campaign for product y, those other products x1 and x2 could be affected, positively or negatively, by promoting product y. For instance, the campaign for y could entice people to visit the site which would be beneficial for all the other products on the site including x1 and x2.

Extend: Try to find additional covariates, two may be too few.

More data: Furthermore, you probably have seasonality in your data, so prediction would probably be helped by learning over a longer time period, such as including several consecutive days or weeks.