# Awful performance of LSTM on noisy time series after stationarisation

Note. The post is quite long because I added some thought process for the sake of seeing the big picture. So grab a coffee and indulge yourself. For tldr the actual question on the bottom.

I put my hands on LSTMs and I was curious of its performance on synthetic data. Thus I decided to see if it predicts well elementary functions. I chose more or less $\log, \exp, \sin$ and identity:

All time series are 1000 long. I decided to train the model so it predicts last 100 elements (i.e. the lag = 100). I've done necessary preprocessing; including data normalisation, but excluding data stationarisation yet. I used Keras Sequential model with the following architecture:

LSTM (300 units) -> Dropout (0.2) -> LSTM (300 units) -> Dropout (0.2) -> Dense (1 unit)

In all cases using batches of size 10 after 10 epochs it performed decently on test set.

$\log, \exp$ and identity serieses are non-stationary, thus to remove trend I stationarised all serieses (I just consider new serieses of differences $\overline{x}_i = x_{i+1} - x_i$), run the training on this new data and reverse modifications.

I was very pleased with the results. After 10 epochs $\log, \exp$ and identity models outrun the models trained on non-stationary data and $\sin$ performed worse (but when I used more epochs it also converged well). For example here are the mean errors of $\log$ function models learned (in 10 epoch each) on non-stationary and stationary data respectively:

Mean error on test set: 3.542939956259637

Mean error on test set: 0.23131714236427667

So far so good, but when I added noise everything starts to crumble. For the sake of concreteness let us constrain to the noisy exp series:

x = np.linspace(0, 10, 1000)
y_exp = 0.01*np.exp(x)
y_exp_noise = 0.01*np.exp(x) + 5*np.random.normal(size = 1000)


y_exp_noise series is clearly non-stationary, so in my opinion it is natural to first transform it to the stationary one before training. However, empirically I found out that after stationarisation the model performs awfully poorly. Here are mean errors of models learned (in 10 epoch each) on non-stationary and stationary data respectively:

Mean error on test set: 6.618303379484316

Mean error on test set: 264.95571226094535

I tried to improve the latter model by grid searching for different batch sizes and by using more epochs, but all was in vain. This noise brakes the models which learns on stationarised data and I don't know why.

Question Why for y_exp_noise series the models learned on stationarised data performs awfully?