LSTM performs poorly with monotonically increasing test set values never seen in training. Why?

Question

I have a dataset of approximately monotonically increasing values (in a time-series). I am using keras and LSTM to train the model and perform the testing on the most recent values in the dataset. For example:

• Training set data from 2009 to 2018
• Test set data form 2018 to 2019 (will have higher values than train set by default )

It just so happens that - due to the increasing nature of the values - the LSTM has never been trained with these large values before. This is making the model perform poorly on new data.

However, when I shuffle the data beforehand i.e. the test set does contain values that the LSTM might have trained on before, the model generalizes better and performs better as well.

1. Is this normal?
2. Is there a way to combat this issue without shuffling?
3. I am using stateless LSTM, so if I standardize the LSTM time windows independently could this be a good solution?

The below is a chart of the dataset: (Not the whole dataset but the large majority of it. Its a good indication of the trend)

Answer 1

You may be suffering from a common issue of neural networks failing to generalize to numerical inputs unseen in training.

The best display of this behavior I know is the figure from this paper:

Caption from the paper: MLPs learn the identity function only for the range of values they are trained on. The mean error ramps up severely both below and above the range of numbers seen during training.

To solve this, you can change the input format of your data so that the higher values aren't "unseen" in the training. You could have it take in and predict relative increases/decreases instead of the absolute value. I think this is what you meant by your third suggestion? ("standardize the time windows").