Time Series forecast with exogenous inputs (present and historic)

Question

I want to model and predict trends in search behavior. To improve the predictive accuracy I want to use "similar trends" and learn from their behavior (from my research this seems to be called an exogenous input). This could be similar behavior at the same point in time (probably easier) but also points in the past. One example might be that the interest in Whitney Houston's death behaves similar to Michael Jackson's death a little bit before that. Is there any machinery that automatically finds these correspondences in time series and uses them? Maybe something that aligns peaks or uses correlation or dynamic time warping to align the sequences the right way for modeling/prediction.

Here are two more examples:

• Google trends for Madden NFL games. One time series has little information about how the future might look like but similar trends (previous games) exhibit very similar behavior.
• Google trends for state football. This is an example that is a little more tricky. Here some trends are somewhat independent over most of the time interval but at the time of the peak they are suddenly very related (football scandal and head coach). It would be great to have some way of automatically figuring out at the beginning of the peak that now I could really use the information of the other time series.

Any ideas and pointers are very welcome. Coding-wise I would prefer to use Python although it seems I might have to learn how to use R. Maybe I could also combine the two (ie. calling R methods from Python).

Answer 1

The problem is similar to forecasting new products by analogy, where the magnitudes of the values in time series may differ, but what you want to compare is the overall shape. Looking at some of that literature could be useful.

A very basic approach, along those lines:

If you have a few different time series that you want to use to model another one, you could choose a time window (e.g. 8 weeks after the event in question) and set the value of each observation to equal the percentage of the total value over the entire window. For example, if after a particular celebrity's death, the first week contains 40% of the activity seen over the entire 8 week period, that would be your first observation; then fill in the percentages for the remaining weeks in the period.

If you do that for all of the relevant time series, you can then compare the overall shape of each; you can disregard any you think are outliers, then apply a simple or weighted average to each period to get an aggregate shape. That can then be used to build a model for the time series you want to forecast.

The process I'm describing is along the lines outlined in this SAS blog post, which may be useful for you to read as well.

I realize I'm not providing any coding help here, but hopefully it's at least a start to how you might approach the problem conceptually.