I am training a 3-layer LSTM on time series data ($10^6$ training samples) to predict the next point in the time series, where there is no seasonality and

  • the time series has been made stationary (using the backshift operator as outlined by Lopez in Advances in Financial Machine Learning - Chapter 5, Fractionally Differentiated Features). Explicitly, if we have a time series $X_t$, then

$$\tilde{X}_t = \sum_{k=0}^\infty \omega_kX_{t-k}$$ where $$\omega = \{ 1, -d, \frac{d(d-1)}{2!}, -\frac{d(d-1)(d-2)}{3!}, ..., (-1)^k\prod_{i=0}^{k-1}\frac{d-i}{k!},... \}$$

where $d$ is the degree of fractional differentiation (I chose $d=0.45$) and I have truncated $\omega$ up to $10^{-5}$. $\tilde{X}_t$ is now the stationary time series. I measure stationarity by the Augmented Dickey Fuller Test; if the ADF statistic is below -2.86, then I categorize the time series as stationary.

  • exponential smoothing has been applied (halflife of 10 periods)
  • the data has been scaled to $[0,1]$

in that order.

All else being left the same, I find that the RMSE on test data is much lower (about 5 times lower) if I do not smooth the data. I can't understand why this would be so.

Intuitively, I would have thought that less noisy data would yield higher accuracy (lower error). In researching resources online, I found this paper where the authors also use an LSTM for time series forecasting and pre-process their data in a similar way (although the order of their pre-processing is different and they apply a simple moving average rather than exponential moving average).

I have two questions:

1. Is there some reason why an LSTM would perform better on noisy data than on smooth data? Is there any literature on this?

2. Does the ordering of preprocessing matter? In other words, should scaling come before smoothing or vice versa? If so, why / why not?

  • 1
    $\begingroup$ "The time series has been made stationary" - a lot of sins can be hidden in those seven words! Could you expand on what you did? $\endgroup$ – jbowman Oct 10 at 19:54
  • $\begingroup$ @jbowman sure, I edited the question to explain how and gave a link to the relevant material $\endgroup$ – PyRsquared Oct 10 at 19:59
  • $\begingroup$ @PyRsquared You just referenced ~400 page document as an explanation of how you made your series stationary. Care to be more specific? $\endgroup$ – Barker Oct 10 at 23:06
  • $\begingroup$ @Barker that's strange; the link was working for me last night but this morning I get the same error as you. Anyway, I've edited the question to include the explicit formula I am using :) $\endgroup$ – PyRsquared Oct 11 at 8:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.