# Proper way of using recurrent neural network for time series analysis

Recurrent neural networks differ from "regular" ones by the fact that they have a "memory" layer. Due to this layer, recurrent NN's are supposed to be useful in time series modelling. However, I'm not sure I understand correctly how to use them.

Let's say I have the following time series (from left to right): [0, 1, 2, 3, 4, 5, 6, 7], my goal is to predict i-th point using points i-1 and i-2 as an input (for each i>2). In a "regular", non-recurring ANN I would do process the data as follows:

 target| input
2| 1 0
3| 2 1
4| 3 2
5| 4 3
6| 5 4
7| 6 5


I would then create a net with two input and one output node and train it with the data above.

How does one need to alter this process (if at all) in the case of recurrent networks?

-

What you describe is in fact a "sliding time window" approach and is different to recurrent networks. You can use this technique with any regression algorithm. There is a huge limitation to this approach: events in the inputs can only be correlatd with other inputs/outputs which lie at most t timesteps apart, where t is the size of the window.

E.g. you can think of a Markov chain of order t. RNNs don't suffer from this in theory, however in practice learning is difficult.

It is best to illustrate an RNN in contrast to a feedfoward network. Consider the (very) simple feedforward network $y = Wx$ where $y$ is the output, $W$ is the weight matrix, and $x$ is the input.

Now, we use a recurrent network. Now we have a sequence of inputs, so we will denote the inputs by $x^{i}$ for the ith input. The corresponding ith output is then calculated via $y^{i} = Wx^i + W_ry^{i-1}$.

Thus, we have another weight matrix $W_r$ which incorporates the output at the previous step linearly into the current output.

This is of course a simple architecture. Most common is an architecture where you have a hidden layer which is recurrently connected to itself. Let $h^i$ denote the hidden layer at timestep i. The formulas are then:

$$h^0 = 0$$ $$h^i = \sigma(W_1x^i + W_rh^{i-1})$$ $$y^i = W_2h^i$$

Where $\sigma$ is a suitable non-linearity/transfer function like the sigmoid. $W_1$ and $W_2$ are the connecting weights between the input and the hidden and the hidden and the output layer. $W_r$ represents the recurrent weights.

Here is a diagram of the structure:

-
I wrong to see some similarity of recurrent networks with Kalman filters? I see this because the previous output affects the present output. What is the practical benefit then of Recurrent networks? – Vass Feb 12 '12 at 21:09
You are write in the sense that both are state space models. However, there are lots of difference: KFs are fully probabilistic, in the sense that the hidden states have a proper probabilistic meaning. RNNs on the other hand are deterministic and only the outputs can be used to model a distribution in a discriminative way. Also, KFs are typically estimated with EM, while RNNs are estimated with gradient based methods. If you want more details, feel free to post a question and send me the link, but the comments are too restricted for this. – bayerj Feb 14 '12 at 18:16
when you connect the output to the input in a recurrent network with a tapped delayed line, isn't that exactly the same as a feed forward network with a sliding window? – siamii Apr 12 '12 at 21:03
No, sliding time window does not pretend on the output of the net, only on the input. – bayerj Apr 12 '12 at 21:36
In OP's question, the output became the input in the next time step, didn't it? for example, target 2 became input 2 in the next step, and target 3, input 3 etc... – siamii Apr 12 '12 at 21:43
show 1 more comment

You may also consider simply using a number of transforms of time series for the input data. Just for one example, the inputs could be:

1. the most recent interval value (7)
2. the next most recent interval value (6)
3. the delta between most recent and next most recent (7-6=1)
4. the third most recent interval value (5)
5. the delta between the second and third most recent (6-5=1)
6. the average of the last three intervals ((7+6+5)/3=6)

So, if your inputs to a conventional neural network were these six pieces of transformed data, it would not be a difficult task for an ordinary backpropagation algorithm to learn the pattern. You would have to code for the transforms that take the raw data and turn it into the above 6 inputs to your neural network, however.

-
You put a lot of domain knowledge into this though. What if you do not recognize the pattern of the time series yourself? How do you then construct a model that can, especially if it depends on inputs that are infinitely far back in the past? – bayerj Mar 10 '11 at 7:01
Infinite would certainly be tricky. However, if you put in transforms of the data that aren't relevant to this domain, the learning algorithm will easily be able to figure that out and adjust the weights accordingly, so it's not a big problem as long as you DO have transforms of the data that are relevant. So, having many different transforms available improves your odds of success. – rossdavidh Mar 10 '11 at 13:45
Imagine the following task: The first input to the net is either $0$ or $1$. Then, the net receives noise from the interval $[-0.1, 0.1]$ for any number (10, 1000, 100000) of timesteps. As soon as it receives $1$ again it has to put out either $0$ or $1$, depending on what it has seen earlier. This is the socalled "Latching benchmark". This is quite a typical setting in sequence learning problems. The big benefit of recurrent networks is, that the whole transformation of inputs itself is learned and NOT given by a human expert or feature engineered. – bayerj Mar 10 '11 at 15:45
I wouldn't wish to say you shouldn't ever use recurrent neural networks; quite the contrary. However, if the task (as stated in the question) is to predict the ith from the (i-1) and (i-2) points, then you can get a better result faster by utilizing that knowledge. I don't mean to suggest that RNN's aren't ever a good idea, but it's ok to use whatever domain knowledge you have to speed up the training process (and decrease the likelihood of training getting caught in a local minimum, etc.). – rossdavidh Mar 11 '11 at 1:59