How to use Machine learning to outperform time series

Question

I am building a ML model to predict future TV audiences based on historical audience data. We currently have a forecasting tool that is using timeseries modelling to make its predictions (mainly ARIMA models). As we want to expand our data input to external factors (weather/channel programming/etc) i'm training a ML model to evaluate our options.

Currently i have the following additional information on top of the audience data:

• weather prediction (temperature, rain/snow, sunshine)
• program category before/after

Based on this info I've created a feature set consisting of a number of lagged timeseries features (audiences week before, month before etc.) plus the weather & programming info. I'm currently using a XGBoost model to evaluate my results and I'm unable to match the performance of the timeseries model based on MSE & Explained variance metrics. (timeseries is performing better)

I'm worried that I'm focusing too much on comparing the performance metrics. I've also tried to introduce log differencing on my y variable and have gotten better performance by doing so, however having to transfer the predictions back to the original state makes me doubt my actions.

As i do not have any experience in replacing timeseries with ML models i was wondering if I'm missing something in my approach here? Also any suggestions for feature creation are welcome.

Thanks in advance for all the help

Answer 1

First off, How to know that your machine learning problem is hopeless? may be enlightening.

The drivers of TV viewership that you are modeling are not very strong.

• Weather may have been a useful predictor forty years ago or more, when there were only two TV channels, so when people were driven inside by bad weather, they only had a choice between these two channels (and reading books, or talking to their family). So the weather effect was concentrated on just two or four different possible outcomes.

  Today, we have hundreds of different TV channels. (Since you are including "program category before/after" as a predictor, it sounds like you are modeling per channel.) Plus people can watch their favorite shows on demand. So if the weather drives them inside, the effect will be distributed across hundreds of possible outcomes, plus people just deciding not to watch TV at all, but catch up on their favorite shows on Netflix, YouTube or other sites.

• "Program category before/after" is also likely not very strong. Yes, of course there is some inertia. But again, with hundreds of channels, the temptation is strong to just flip through the other channels to see what is there, instead of waiting through the inevitable commercial break to see what will come on the channel I have been watching next. Again, sixty years ago, TV sets had to be changed by getting up from your couch - but today, remote controls mean we don't even have to get up any more.

Bottom line: you are modeling very weak predictors. This will only work if you have * a lot* of data. (I gave an illustration in Kolassa, 2016, Foresight.) And you don't have that here. Even if your history goes back multiple years, the data generating process has changed, simply because alternatives like YouTube and Netflix were vastly less important a few years ago.

In a situation like this, simple time series algorithms are very hard to beat.

The best way forward, per the link at the very top, is to identify drivers you have not thought of yet. Not being a TV person, I can't help you there, with one exception: you almost certainly have multiple-seasonalities, namely diurnal patterns that are different on the weekend than during the week. So my recommendation would be to look at methods that were designed to model such multiple seasonalities, as an alternative to ARIMA. The tag wiki contains more information and pointers.

Answer 2

I have gone through this before as well so I'll share a few learnings we had upgrading from traditional stat time series to boosted trees:

Feeding lagged target features to xgboost/lightgbm typically didn't work and added a lot of complexity due to needing to predict multiple times. Calendar features such as week number/ month number could cause a lot of overfitting especially if you have shorter training history.

So some things that helped us a lot:

We are forecasting at multiple levels such as product/country so doing standard scaling at each level helped tremendously. At each product/country or whatever you are forecasting for pass simple model outputs as features such as the mean, maybe a more localized measure of mean/ ets, a simple measure for seasonality, a simple linear trend, a more local measure for trend/ ets. Also pass some time series features as well such as average gradient, skewness, entropy, lumpiness, kpss, stability, etc. Hyndman and the team which submitted an ensemble for the M4 competition did a lot of great work here. We also did several runs to have a ton of data to learn from using both the fitted results from those simple models as well as forecast results from them. Category/ID features especially helped for low history items. You can supplement your data further with some simulated data just be careful that the model won't learn bad things!

Overall this approach was a big improvement over traditional stat and other ML such as LSTM, but definitely takes some work.