Proper way of using recurrent neural network for time series analysis

Question

Recurrent neural networks differ from "regular" ones by the fact that they have a "memory" layer. Due to this layer, recurrent NN's are supposed to be useful in time series modelling. However, I'm not sure I understand correctly how to use them.

Let's say I have the following time series (from left to right): [0, 1, 2, 3, 4, 5, 6, 7], my goal is to predict i-th point using points i-1 and i-2 as an input (for each i>2). In a "regular", non-recurring ANN I would do process the data as follows:

 target| input
      2| 1 0
      3| 2 1
      4| 3 2
      5| 4 3
      6| 5 4
      7| 6 5 

I would then create a net with two input and one output node and train it with the data above.

How does one need to alter this process (if at all) in the case of recurrent networks?

Answer 1

What you describe is in fact a "sliding time window" approach and is different to recurrent networks. You can use this technique with any regression algorithm. There is a huge limitation to this approach: events in the inputs can only be correlatd with other inputs/outputs which lie at most t timesteps apart, where t is the size of the window.

E.g. you can think of a Markov chain of order t. RNNs don't suffer from this in theory, however in practice learning is difficult.

It is best to illustrate an RNN in contrast to a feedfoward network. Consider the (very) simple feedforward network $y = Wx$ where $y$ is the output, $W$ is the weight matrix, and $x$ is the input.

Now, we use a recurrent network. Now we have a sequence of inputs, so we will denote the inputs by $x^{i}$ for the ith input. The corresponding ith output is then calculated via $y^{i} = Wx^i + W_ry^{i-1}$.

Thus, we have another weight matrix $W_r$ which incorporates the output at the previous step linearly into the current output.

This is of course a simple architecture. Most common is an architecture where you have a hidden layer which is recurrently connected to itself. Let $h^i$ denote the hidden layer at timestep i. The formulas are then:

$$h^0 = 0$$ $$h^i = \sigma(W_1x^i + W_rh^{i-1})$$ $$y^i = W_2h^i$$

Where $\sigma$ is a suitable non-linearity/transfer function like the sigmoid. $W_1$ and $W_2$ are the connecting weights between the input and the hidden and the hidden and the output layer. $W_r$ represents the recurrent weights.

Here is a diagram of the structure:

Answer 2

You may also consider simply using a number of transforms of time series for the input data. Just for one example, the inputs could be:

1. the most recent interval value (7)
2. the next most recent interval value (6)
3. the delta between most recent and next most recent (7-6=1)
4. the third most recent interval value (5)
5. the delta between the second and third most recent (6-5=1)
6. the average of the last three intervals ((7+6+5)/3=6)

So, if your inputs to a conventional neural network were these six pieces of transformed data, it would not be a difficult task for an ordinary backpropagation algorithm to learn the pattern. You would have to code for the transforms that take the raw data and turn it into the above 6 inputs to your neural network, however.

Answer 3

Another possibility are Historical Consistent Neural Networks (HCNN). This architecture might be more appropriate for the above mentioned setup because they eliminate the often arbitrary distinction between input- and output-variables and instead try to replicate the full underlying dynamics of the whole system via training with all observables.

When I was working for Siemens I published a paper on this architecture in a book by Springer Verlag:

Zimmermann, Grothmann, Tietz, von Jouanne-Diedrich: Market Modeling, Forecasting and Risk Analysis with Historical Consistent Neural Networks.

Just to give an idea about the paradigm here is a short excerpt:

In this article, we present a new type of recurrent NN, called historical consistent neural network (HCNN). HCNNs allow the modeling of highly-interacting non-linear dynamical systems across multiple time scales. HCNNs do not draw any distinction between inputs and outputs, but model observables embedded in the dynamics of a large state space.

[...]

The RNN is used to model and forecast an open dynamic system using a non-linear regression approach. Many real-world technical and economic applications must however be seen in the context of large systems in which various (non-linear) dynamics interact with each other in time. Projected on a model, this means that we do not differentiate between inputs and outputs but speak about observables. Due to the partial observability of large systems, we need hidden states to be able to explain the dynamics of the observables. Observables and hidden variables should be treated by the model in the same manner. The term observables embraces the input and output variables (i. e. $Y_τ := (y_τ, u_τ)$). If we are able to implement a model in which the dynamics of all of the observables can be described, we will be in a position to close the open system.

...and from the conclusion:

The joint modeling of hidden and observed variables in large recurrent neural networks provides new prospects for planning and risk management. The ensemble approach based on HCNN offers an alternative approach to forecasting of future probability distributions. HCNNs give a perfect description of the dynamic of the observables in the past. However, the partial observability of the world results in a non-unique reconstruction of the hidden variables and thus, different future scenarios. Since the genuine development of the dynamic is unknown and all paths have the same probability, the average of the ensemble may be regarded as the best forecast, whereas the bandwidth of the distribution describes the market risk. Today, we use HCNN forecasts to predict prices for energy and precious metals to optimize the timing of procurement decisions. Work currently in progress concerns the analysis of the properties of the ensemble and the implementation of these concepts in practical risk management and financial market applications.

The paper is now finally available in full here: Zimmermann, Grothmann, Tietz, von Jouanne-Diedrich: Market Modeling, Forecasting and Risk Analysis with Historical Consistent Neural Networks.