For example:

I want to forecast future values of a time-series based on previous values of multiple time-series' using a ANN and/or SVM. Inputs will be lagged values from each time series, and the outputs will be one-step-ahead forecasts (forecasts with further horizons will be done by "rolling" the predictions forward using previous predictions).

Now, shouldn't SVMs and ANNs be able to learn trends and cycles? Would they not be able to learn things like, "with all else being equal the output of this series should be 2x the previous output?" Or, if I provide a categorical variable for month, "since it's January, divide the prediction I would've made by 2?"

Would attempting to decycle and detrend the data result in imposing more bias than necessary?

  • 1
    $\begingroup$ From the stand point of machine learning and prediction, try it multiple ways and see what works best - i.e. yields the best prediction. $\endgroup$ – B_Miner Jun 29 '12 at 18:33

With machine learning algorithms it is often beneficial to use feature scaling or normalisation to help the algorithm converge quickly during the training and to avoid one set of features dominating another. Take, for example, the problem of predicting stock prices. If you include high priced stocks such as Apple or Microsoft along with some penny stocks, the high valued features you will necessarily extract from Apple and Microsoft prices will overwhelm those that you extract from the penny stocks, and you won't be training on an apple to apple basis ( no pun intended! ), and the resultant trained model might not generalise very well.

However, imho "attempting to decycle and detrend the data" would be a very good thing to do. Extracting the various cyclic and trend components and normalising them by subtracting their respective means and dividing by their standard deviations would place all the data for all time series into the same approximate range, and then you would be training on like to like data which, when rescaled by reversing the normalisation, would likely generalise much better for predictive purposes.

Furthermore, for any time series it might be the case that trend swamps the cyclic component, so you might end up training on trend only data which almost certainly won't perform well on cyclic time series, and vice versa. By separating out the two components and training on each with separate SVMs or NNs and then recombining the two predictions, you might end up with a more accurate and more readily generalisable algorithm.

  • $\begingroup$ Yeah, forgot that most (all?) standardization and regularization methods assume stationary distributions. Separating components out and building additive models is a good tip too. $\endgroup$ – ektrules Jul 1 '12 at 22:02
  • $\begingroup$ Though, now I'm wondering about the loss of information that accompanies stationary transformations. What about when the actual non-differenced values matter? For instance, if you have a time series of debt and GDP, then difference both, the ML method can only make learn the effects of the changes, and the effects may be different if debt is 10% of GDP vs 200%. (of course, you can add a GDP/Debt feature). $\endgroup$ – ektrules Jul 1 '12 at 22:14

How far ahead are you predicting compared to the timescales that trend or cycles operate on? Zhang, Qi 2005 - 'Neural network forecasting for seasonal and trend time series' find de-seasonalising and de-trending (DSDT) beneficial, but their prediction timescales are similar to their trend/seasonal timescales. In contrast I've been working on data where I make short timescale predictions (e.g. 1 day) and trend/seasonality only act over much longer timescales. DSDT does still improve my predictive accuracy to some degree, but the ML can cope reasonably well on its own without DSDT as trend/seasonality is effectively irrelevant for the last few data points.


I'm pretty sure you are using wrong tools here.

ML methods are created for interpolation (like predicting time series A from time series B and C); for extrapolations we have Markov chains and friends.

The problem with your approach is that it is terribly easy to overfit the model in this conditions and, what's worse, it is hard to spot this (normal cross-validation will fail, so it is very hard to fit parameters the proper way, etc.).
Adding explicit time to predictors is also a bad idea -- I have seen models fitted only on time and decision with 90% accuracy on cross-validation and random guessing on post-training-data tests. If you need time, it is better to include it as a series of cycle descriptors like day of week or seconds past midnight, obviously never exceeding or even going near the length of your training series.

  • $\begingroup$ You may be right about using the wrong tools, IDK. I haven't really found any online materials using the same procedure as I'm thinking of. I actually am trying to predict a time-series from multiple other time-series (over the same period), but I'd like to predict into the future as well (what good is it to predict a value when it's already available at the time of prediction?). To do the rolling predictions, I'll actually have to predict every time-series though. I guess what I'm trying to do is something like Vector Autoregression, but with ML approaches. Will this approach not work well? $\endgroup$ – ektrules Jun 30 '12 at 20:08
  • $\begingroup$ Why does the time as a predictor variable (to forecast the trend rather than cyclicality) work so poorly? $\endgroup$ – max Feb 6 '16 at 2:22
  • 1
    $\begingroup$ As I wrote: 1/ ML is bad at extrapolation, 2/ time uniquely identifies each object, which allows severe overfitting and inhibits pursuit for actual interactions. $\endgroup$ – user88 Feb 7 '16 at 12:26
  • 1
    $\begingroup$ @mbq For (1), I thought only local methods are terrible at extrapolating, while global methods might be alright. But I can see how time series / MC might be more appropriate. For (2) though, I disagree. Using a unique identifier would lead to a good training score but to a horrible CV score. So you should be able to spot bad models. The problem I think is that you might still overfit somewhat with traditional CV since you use future time. Perhaps a train-test split by time, with test at the end, would make more sense. $\endgroup$ – max Jul 17 '16 at 16:41
  • $\begingroup$ @max Yes, I was thinking about CV that splits moments randomly; using continuous time blocks will help. $\endgroup$ – user88 Jul 18 '16 at 11:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.