I have seen many kernels that are using machine learning algorithms (Xgb, LSTM, others) on time series forecasting.

A time series data typically has trend and seasonal components. In general my question is

  1. Is it necessary to remove trend and seasonality (i.e make it stationary ) before applying machine learning/supervised learning (Xgb, LSTM, others) algorithms for time series data ?

  2. when will machine learning/supervised learning (Xgb, LSTM, others) algorithms for time series data gives good result? When will they not give good result?

  3. Any guidelines for using machine learning/supervised learning algorithms for time series data ?

  4. If there is seasonality and trend how to tackle the problem? One way is to detrend and remove seasonality, and then use ML algorithm for forecasting.
    Are there any other approaches especially if time series has trend?

  5. Finally, How will you verify the forecast results? I mean, can we look at the residuals/something to infer forecast results makes sense.

  • $\begingroup$ Your should definitly check freerangestats.info/blog/2016/11/06/forecastxgb $\endgroup$ – Fabian Werner Jan 7 at 14:48
  • $\begingroup$ As some more information in machine learning approaches vs. traditional statistical ones, the very recent M4 competition just closed this year and I highly encourage you to take a read on the results: researchgate.net/publication/… In summary, all "pure" ML approaches did not perform well at all. Out of the top five models, I believe four were ensembles of classical methods, and first place was a hybrid neural network + Holt-Winters multiplicative $\endgroup$ – aranglol Jan 7 at 20:30
  1. "ML" approaches will typically do better in high signal to noise situations, and with enough data. With less data, the traditional forecasting approaches are often superior.

  2. This is a very broad question. One recommendation: familiarize yourself with the state of the art in "classical" forecasting. I recommend the excellent free online book Forecasting: Principles and Practice (2nd ed.) by Athanasopoulos & Hyndman. Test your ML forecasts against simple benchmarks, which surprisingly often outperform more complex algorithms.

  3. One way to deal with trend is to include a linearly increasing predictor and extrapolate that out. Forecasters often find that dampening the trend improves forecasts. Similarly, you can include periodic functions like harmonics or bump functions as predictors to model seasonality.

  4. There are many accepted measures of forecast accuracy. You may also want to look through the tag wikis on the MAPE, the MASE, the MAE or the MSE, or through threads tagged both "forecasting" and "accuracy".

  • $\begingroup$ Can you explain 3? $\endgroup$ – vijju Jan 7 at 16:50
  • $\begingroup$ For trend, include a predictor that counts the seconds, days or whatever time granularity you have, starting at an arbitrary beginning of time. Both historically and for the future. For seasonality, transform this into multiple periodic functions whose periodicity is a multiple of your seasonality base periodicity. E.g., for intra-yearly seasonality on daily data, include one sine wave with period 365.25, one with 182.6, one with 121.8 etc. $\endgroup$ – Stephan Kolassa Jan 7 at 16:58
  • $\begingroup$ Thanks for reply. According to the the above comment, it is not always mandatory to make your time series stationary before applying ml algorithm to time series forecasting. Am I right? $\endgroup$ – vijju Jan 7 at 17:43
  • $\begingroup$ @aranglol: thank you. I am aware of the M4 competition. (Right now, I happen to be working on a commentary that the guest editor of the IJF invited me to write.) Your summary is correct, but the question really is how informative the M4 setup was. ML approaches work better with (lots of) causal and meta data, and this was not available in the M4. I have heard that some ML experts declined to participate for this exact reason. I am by no means a proponent of ML, but the M4 provides only a very weak argument against it. $\endgroup$ – Stephan Kolassa Jan 7 at 19:29
  • $\begingroup$ Stationarity is a tricky thing. "Plain vanilla" ML implementations, e.g., LSTMs, that assume stationarity will of course require stationarity. But it's straightforward to model certain kinds of nonstationarity without transforming the input. For instance, you can model trends and seasonalities - two kinds of nonstationarity - in the way I described. So the question is really what assumptions your algorithm of choice makes. $\endgroup$ – Stephan Kolassa Jan 7 at 19:32

In my experience, LSTM tends to be more effective when it comes to a time series that shows significant volatility - it is designed to handle sequence data and for this reason is quite flexible at adapting to sudden changes in a time series.

Let's take an example. Suppose we are interested in forecasting electricity consumption for a commercial building. We can see that the data is quite volatile:


Let's see how LSTM compares to ARIMA in making forecasts:





We can see that the predictions on LSTM follow the actual time series for the most part, whereas the confidence interval for ARIMA is particularly wide.

ARIMA tends to be better when it comes to linear relationships, and while LSTM is still able to model these, it can be somewhat of an overkill to implement this. Unless your data is significantly volatile, ARIMA should do a good job of modelling.

When it comes to forecasting time series (whether with ARIMA or LSTM), it is important to ensure stationarity in the model (by differencing if necessary).

The reason for this is that a non-stationary time series has an inconsistent mean, variance, and autocorrelation, and this makes forecasting more difficult. In most cases, doing this should remove any trends or stationarity. If not, taking a higher-order difference (e.g. 2nd, 3rd differences etc) might be necessary.

  • 1
    $\begingroup$ If I understand your analysis correctly (only having skimmed your linked page), the LSTM uses explanatory variables. I assume these account for the sharp downward spikes. Conversely, the ARIMA model does not use explanatory variables in either an ARIMAX model or a regression with ARIMA errors, right? How would an ARIMAX model fare? Or, even better a (T)BATS model that could account for the multiple-seasonalities likely present (intra-weekly and intra-yearly)? $\endgroup$ – Stephan Kolassa Jan 7 at 19:37
  • $\begingroup$ No, LSTM does not use explanatory variables - this was another example of a standard neural network. The sharp downward spikes are endemic to the time series itself. ARIMAX could potentially fare well but it's not a given. I've seen situations where a pure ARIMA model could predict the trend better than ARIMAX with explanatory variables. I haven't used the TBATS model specifically, but have used the Kalman Filter for time series modelling, which is also quite effective at adapting to changes quickly in a similar manner to LSTM. $\endgroup$ – Michael Grogan Jan 8 at 11:08
  • 2
    $\begingroup$ @StephanKolassa zoom in on the LSTM plot and you will notice that it is really just a random walk with only 1 step ahead forecasts (a common mistake Im noticing with people trying to use DL for forecasting), whereas the ARIMA model is doing multi step forecasting, hence the comparison is not fair. $\endgroup$ – Skander H. Jan 15 at 16:33
  • $\begingroup$ @MichaelGrogan: I also think that this is a wrong plot, as Skander H. comments. The LSTM does an one-step-ahead forecasts over a 94 steps while the ARIMA is doing a single 200+ steps forecast. Maybe the LSTM is indeed a competitive alternative but no evidence has been provided to support this claim in example shown. $\endgroup$ – usεr11852 Feb 22 at 22:56
  • $\begingroup$ I've actually done further investigation on this in the past month (my analysis is at michaeljgrogan.com/electricity-consumption-neural). Essentially, I found that with stationary data, LSTM can actually function quite well over longer time periods such as 50 days. I accept the comparison above isn't an "apples to apples" one, but the test error for this particular example still remained reasonably low even as the time period was increased. $\endgroup$ – Michael Grogan Feb 22 at 23:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.