3
$\begingroup$

I've been searching for a while now to find the precise way to feed a Recurrent Neural Network (RNN, LSTM, GRU, ESN, Etc) with time series data with no real success.

Here is a question that was close, but the answers aren't very clear: Proper way of using recurrent neural network for time series analysis

I'm not looking for a breakdown of how the networks work, but rather how to structure the input/output vectors for optimal results.

So, let's say I'm working with a growing sinusoid. Here is my data set with 50 values:

{ 40, 43.545, 47.214, 50.518, 52.959, 54.102, 53.639, 51.449, 47.621, 42.47, 36.506, 30.392, 24.864, 20.646, 18.356, 18.424, 21.022, 26.027, 33.015, 41.295, 49.977, 58.067, 64.579, 68.656, 69.681, 67.358, 61.772, 53.394, 43.044, 31.816, 20.959, 11.741, 5.299, 2.5, 3.833, 9.327, 18.537, 30.57, 44.172, 57.857, 70.071, 79.368, 84.577, 84.954, 80.276, 70.889, 57.687, 42.03, 25.605, 10.242 };

enter image description here

I am very familiar with the sliding time window approach that works well with Feed forward networks (FFN). Normalization aside, I would do something like this:

Train with 49 of the 50 values and validate with the last
3 inputs, 1 output - Sliding Time Window

Train:

    Input                     Output
-------------------------------------
[1] 40, 43.545, 47.214     -> 50.518
[2] 43.545, 47.214, 50.518 -> 52.959
...
[49] 70.889, 57.687, 42.03 -> 25.605

Validate:

     Input                     Output
-------------------------------------
[50] 57.687, 42.03, 25.605  -> 10.242

And this works very well with FFN (And with RNN also), but I'm led to believe RNN shouldn't need to be setup that way. Rather they work very well with time series data and all I should need need to do is have the input be the previous output. Something like this:

Train with 49 of the 50 values and validate with the last
1 input, 1 output

Train:

    Input     Output
-------------------------------------
[1] 40     -> 43.545
[2] 43.545 -> 47.214
...
[49] 42.03 -> 25.605

Validate:

     Input    Output
-------------------------------------
[50] 25.605 -> 10.242

But this has produced horrible results with my RNNs. I can't seem to get a handle on how to feed time series data. I've tried the input being the index, such as:

    Input     Output
-------------------------------------
[1] 1      -> 40     
[2] 2      -> 43.545
[3] 3      -> 47.214

I've even tried combinations of this and the previous setup and nothing is working well.

Could someone please show me exactly how the input and output vectors should be for this scenario?

$\endgroup$
  • $\begingroup$ Check out this paper for some nice examples: felixgers.de/papers/phd.pdf $\endgroup$ – BHP May 4 '16 at 21:58
  • $\begingroup$ What is the size of the hidden layer ? Are you sure that your hyper-parameters are well-tuned ? Are you using LTSM or something else ? $\endgroup$ – Romain May 5 '16 at 0:10
  • $\begingroup$ I think you should set up your RNN with timestep = 3. If you want memory to pass through each iteration you need your RNN to be stateful. $\endgroup$ – Akylas Stratigakos Dec 13 '18 at 10:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.