2
$\begingroup$

When training an RNN for time series prediction, what can one expect to see visually as the model learns? In particular, are plateaus a normal indication that the model is underfitting or do they represent a more fundamental problem with the algorithm?

In my case, I am training a Keras RNN to predict a (partly noisy) time series using 30000 training examples (namely, sliding windows). The result looked like this on the training set

Training set with 30000 training examples

and like this on the testing set

Testing set with 30000 training examples

To me, the above looks promising: The prediction on the training set is predictably tighter than that on the testing set, even if the latter still manages to capture the overall trend.

However, if I only train on 300 training samples, the training set looks like this

Training set with 300 training examples

and the testing looks like

Testing set with 300 training examples.

I would expect that because I only trained on 300 samples instead of 300000, the performance would be worse. This is indeed the case. But, how about the plateaus? Can I conclude that they arise as a direct consequence of the fact that there aren't enough training samples? I.e., that the RNN, for lack of data, just picks the previous value as the predicted value on certain segments of the test sets?

PS: I noticed similar plateaus when I drastically decrease the number of hidden layers. Once again, it seems to make sense since that implies the model is heavily biased. It's just that the plateaus show up at seemingly random segments of the test set---and I can't really make sense of that.

Improve this question
$\endgroup$
4
  • $\begingroup$ A common mistake when modeling time-series data with RNNs is forgetting to take the first differences. Yesterday's temperature/stock price/approval rating is usually a great predictor of today's, so your RNN could be learning to merely repeat the last-seen value. I'm wondering if that explains why your predictions in the training plot look like the the true values, but offset by 1 time step. $\endgroup$
    – Sycorax ♦
    Commented Aug 14, 2018 at 13:40
  • $\begingroup$ @Sycorax From the looks of it, this isn't what's happening here. The RNN is fed with windows, not simply the current value. Besides, I don't see how that explains the plateaus. $\endgroup$
    – Tfovid
    Commented Aug 14, 2018 at 14:03
  • 1
    $\begingroup$ Are you performing 1 step or multi step forecasts? $\endgroup$
    – Skander H.
    Commented Aug 14, 2018 at 15:17
  • $\begingroup$ @Alex I am performing a 1-step forecast. $\endgroup$
    – Tfovid
    Commented Aug 15, 2018 at 9:50

1 Answer 1

Reset to default
1
$\begingroup$

(More of an extended comment than a real answer):

I tend to agree with Sycorax: A common issue with RNN is that they end up simulating a random walk (i.e. your best bet is to use the most recent value as your next step forecast), and this seems to be the behavior shown in your 30000 training sample plot. Your RNN simply memorized the training data and is returning a naive one step ahead forecast.

This also seems to be confirmed by the 3rd plot you are showing, the one of the training set with only 300 samples: The RNN has memorized only parts of the data and not all of it and the plateaus correspond to the parts that it hasn't memorized yet.

As I said this is more of comment. To get a definitive answer, you should try the experiment 300 vs. 30000 samples on the same data set (you seem to be using two different series here), and also give an indication of the input and output window sizes you are using and how you are batching the data.

Improve this answer
$\endgroup$
5
  • $\begingroup$ I have two follow-up questions: (1) Why would the RNN only memorize random segments of the training set and not others, thereby leading to the plateaus? (2) Assuming that my RNN is only learning to reproduce the data with a shift of one step, what can I do to fix the problem? PS: I have a window size of 6 time units. As for the output, I'm only predicting the next value, not a whole look-forward window. $\endgroup$
    – Tfovid
    Commented Aug 14, 2018 at 16:02
  • $\begingroup$ @Tfovid (1) It depends on how you are batching your data. (2) As Sycorax said, you could use differencing. A Box-Cox transformation could help as well. I succeeded in getting good forecasts with RNN-LSTM (on a data set that was less noisy than yours) by doing normalization $(data - mean) / variance$. $\endgroup$
    – Skander H.
    Commented Aug 14, 2018 at 17:24
  • $\begingroup$ I'm a bit confused as to how one can tell whether the algo is merely memorizing the previous step or doing some genuine prediction. Consider a zoomed-in segment of the training and test sets. (I don't just get a shift in the time series.) $\endgroup$
    – Tfovid
    Commented Aug 15, 2018 at 8:29
  • $\begingroup$ ... as for the batching I use batches of 72 data samples, namely 72 sliding windows of 6 time units each. $\endgroup$
    – Tfovid
    Commented Aug 15, 2018 at 8:42
  • 1
    $\begingroup$ "how one can tell whether the algo is merely memorizing the previous step or doing some genuine prediction." - memorizing is more or less the same thing as overfitting. If you training error is very small but your test error is large, you can tell that your RNN isn't really predicting, just memorizing. $\endgroup$
    – Skander H.
    Commented Aug 15, 2018 at 22:20

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.