66
$\begingroup$

Recurrent neural networks differ from "regular" ones by the fact that they have a "memory" layer. Due to this layer, recurrent NN's are supposed to be useful in time series modelling. However, I'm not sure I understand correctly how to use them.

Let's say I have the following time series (from left to right): [0, 1, 2, 3, 4, 5, 6, 7], my goal is to predict i-th point using points i-1 and i-2 as an input (for each i>2). In a "regular", non-recurring ANN I would do process the data as follows:

 target| input
      2| 1 0
      3| 2 1
      4| 3 2
      5| 4 3
      6| 5 4
      7| 6 5 

I would then create a net with two input and one output node and train it with the data above.

How does one need to alter this process (if at all) in the case of recurrent networks?

$\endgroup$
  • $\begingroup$ Have you found out how to structure the data for the RNN (e.g. LSTM)? thank you $\endgroup$ – mik1904 Sep 13 '17 at 13:30
49
$\begingroup$

What you describe is in fact a "sliding time window" approach and is different to recurrent networks. You can use this technique with any regression algorithm. There is a huge limitation to this approach: events in the inputs can only be correlatd with other inputs/outputs which lie at most t timesteps apart, where t is the size of the window.

E.g. you can think of a Markov chain of order t. RNNs don't suffer from this in theory, however in practice learning is difficult.

It is best to illustrate an RNN in contrast to a feedfoward network. Consider the (very) simple feedforward network $y = Wx$ where $y$ is the output, $W$ is the weight matrix, and $x$ is the input.

Now, we use a recurrent network. Now we have a sequence of inputs, so we will denote the inputs by $x^{i}$ for the ith input. The corresponding ith output is then calculated via $y^{i} = Wx^i + W_ry^{i-1}$.

Thus, we have another weight matrix $W_r$ which incorporates the output at the previous step linearly into the current output.

This is of course a simple architecture. Most common is an architecture where you have a hidden layer which is recurrently connected to itself. Let $h^i$ denote the hidden layer at timestep i. The formulas are then:

$$h^0 = 0$$ $$h^i = \sigma(W_1x^i + W_rh^{i-1})$$ $$y^i = W_2h^i$$

Where $\sigma$ is a suitable non-linearity/transfer function like the sigmoid. $W_1$ and $W_2$ are the connecting weights between the input and the hidden and the hidden and the output layer. $W_r$ represents the recurrent weights.

Here is a diagram of the structure:

schematic

$\endgroup$
  • 2
    $\begingroup$ I wrong to see some similarity of recurrent networks with Kalman filters? I see this because the previous output affects the present output. What is the practical benefit then of Recurrent networks? $\endgroup$ – Vass Feb 12 '12 at 21:09
  • 8
    $\begingroup$ You are write in the sense that both are state space models. However, there are lots of difference: KFs are fully probabilistic, in the sense that the hidden states have a proper probabilistic meaning. RNNs on the other hand are deterministic and only the outputs can be used to model a distribution in a discriminative way. Also, KFs are typically estimated with EM, while RNNs are estimated with gradient based methods. If you want more details, feel free to post a question and send me the link, but the comments are too restricted for this. $\endgroup$ – bayerj Feb 14 '12 at 18:16
  • 1
    $\begingroup$ No, sliding time window does not pretend on the output of the net, only on the input. $\endgroup$ – bayerj Apr 12 '12 at 21:36
  • 2
    $\begingroup$ @bayerj great information, but i don't think you answered the question. How do you structure the input output vectors not in a sliding time window for RNNs? Can you provide a couple samples with the OP's dataset? $\endgroup$ – Levitikon Dec 16 '15 at 19:10
  • 1
    $\begingroup$ This is a very informative description of RNNs but I fail to find an answer to OP's question: How does one need to alter [training] in the case of recurrent networks? $\endgroup$ – wehnsdaefflae Jun 22 '16 at 12:22
9
$\begingroup$

You may also consider simply using a number of transforms of time series for the input data. Just for one example, the inputs could be:

  1. the most recent interval value (7)
  2. the next most recent interval value (6)
  3. the delta between most recent and next most recent (7-6=1)
  4. the third most recent interval value (5)
  5. the delta between the second and third most recent (6-5=1)
  6. the average of the last three intervals ((7+6+5)/3=6)

So, if your inputs to a conventional neural network were these six pieces of transformed data, it would not be a difficult task for an ordinary backpropagation algorithm to learn the pattern. You would have to code for the transforms that take the raw data and turn it into the above 6 inputs to your neural network, however.

$\endgroup$
  • 1
    $\begingroup$ You put a lot of domain knowledge into this though. What if you do not recognize the pattern of the time series yourself? How do you then construct a model that can, especially if it depends on inputs that are infinitely far back in the past? $\endgroup$ – bayerj Mar 10 '11 at 7:01
  • $\begingroup$ Infinite would certainly be tricky. However, if you put in transforms of the data that aren't relevant to this domain, the learning algorithm will easily be able to figure that out and adjust the weights accordingly, so it's not a big problem as long as you DO have transforms of the data that are relevant. So, having many different transforms available improves your odds of success. $\endgroup$ – rossdavidh Mar 10 '11 at 13:45
  • $\begingroup$ Imagine the following task: The first input to the net is either $0$ or $1$. Then, the net receives noise from the interval $[-0.1, 0.1]$ for any number (10, 1000, 100000) of timesteps. As soon as it receives $1$ again it has to put out either $0$ or $1$, depending on what it has seen earlier. This is the socalled "Latching benchmark". This is quite a typical setting in sequence learning problems. The big benefit of recurrent networks is, that the whole transformation of inputs itself is learned and NOT given by a human expert or feature engineered. $\endgroup$ – bayerj Mar 10 '11 at 15:45
  • 2
    $\begingroup$ I wouldn't wish to say you shouldn't ever use recurrent neural networks; quite the contrary. However, if the task (as stated in the question) is to predict the ith from the (i-1) and (i-2) points, then you can get a better result faster by utilizing that knowledge. I don't mean to suggest that RNN's aren't ever a good idea, but it's ok to use whatever domain knowledge you have to speed up the training process (and decrease the likelihood of training getting caught in a local minimum, etc.). $\endgroup$ – rossdavidh Mar 11 '11 at 1:59
4
$\begingroup$

Another possibility are Historical Consistent Neural Networks (HCNN). This architecture might be more appropriate for the above mentioned setup because they eliminate the often arbitrary distinction between input- and output-variables and instead try to replicate the full underlying dynamics of the whole system via training with all observables.

When I was working for Siemens I published a paper on this architecture in a book by Springer Verlag: Zimmermann, Grothmann, Tietz, von Jouanne-Diedrich: Market Modeling, Forecasting and Risk Analysis with Historical Consistent Neural Networks

Just to give an idea about the paradigm here is a short excerpt:

In this article, we present a new type of recurrent NN, called historical consistent neural network (HCNN). HCNNs allow the modeling of highly-interacting non-linear dynamical systems across multiple time scales. HCNNs do not draw any distinction between inputs and outputs, but model observables embedded in the dynamics of a large state space.

[...]

The RNN is used to model and forecast an open dynamic system using a non-linear regression approach. Many real-world technical and economic applications must however be seen in the context of large systems in which various (non-linear) dynamics interact with each other in time. Projected on a model, this means that we do not differentiate between inputs and outputs but speak about observables. Due to the partial observability of large systems, we need hidden states to be able to explain the dynamics of the observables. Observables and hidden variables should be treated by the model in the same manner. The term observables embraces the input and output variables (i. e. $Y_τ := (y_τ, u_τ)$). If we are able to implement a model in which the dynamics of all of the observables can be described, we will be in a position to close the open system.

...and from the conclusion:

The joint modeling of hidden and observed variables in large recurrent neural networks provides new prospects for planning and risk management. The ensemble approach based on HCNN offers an alternative approach to forecasting of future probability distributions. HCNNs give a perfect description of the dynamic of the observables in the past. However, the partial observability of the world results in a non-unique reconstruction of the hidden variables and thus, different future scenarios. Since the genuine development of the dynamic is unknown and all paths have the same probability, the average of the ensemble may be regarded as the best forecast, whereas the bandwidth of the distribution describes the market risk. Today, we use HCNN forecasts to predict prices for energy and precious metals to optimize the timing of procurement decisions. Work currently in progress concerns the analysis of the properties of the ensemble and the implementation of these concepts in practical risk management and financial market applications.

Parts of the paper can be viewed publicly: Here

$\endgroup$
  • $\begingroup$ Do you have an implementation available to download and test? $\endgroup$ – Julien L Jun 1 '16 at 13:44
  • $\begingroup$ @JulienL: Unfortunately not because this was proprietary work for Siemens. $\endgroup$ – vonjd Jun 1 '16 at 13:53
  • $\begingroup$ Too bad, that looked promising. $\endgroup$ – Julien L Jun 2 '16 at 15:29
  • $\begingroup$ @JulienL: I encourage you to contact Georg, my co-author. His email is on the first page of the paper (see link above). $\endgroup$ – vonjd Jun 2 '16 at 18:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.