Difference between Time delayed neural networks and Recurrent neural networks

Question

I would like to use a Neural Network to predict financial time series. I come from an IT background and have some knowledge of Neural Networks and I have been reading about these:

• TDNN
• RNN

I have been searching for R packages for them and I only found one for RNN, the RSNNS package which has elman and jordan implementations which are RNN.

So, are Recurrent Neural Networks useful to use with (financial) time series? Since they (quote from the wikipedia link on RNN cited before):

At each time step, the input is propagated in a standard feed-forward fashion, and then a learning rule is applied. The fixed back connections result in the context units always maintaining a copy of the previous values of the hidden units (since they propagate over the connections before the learning rule is applied). Thus the network can maintain a sort of state, allowing it to perform such tasks as sequence-prediction that are beyond the power of a standard multilayer perceptron.

aren't in practice the same as Time Delay Neural Networks? If not, which are the differences with Time Delay Neural Networks? Are both suitable to use with Time Series or which one is more suitable?

Thanks beforehand!

Answer 1

I have never worked with recurrent networks, but from what I know, in practice, some RNN and TDNN can be used for the same purpose that you want: Predict time series values. However, they work different.

It is possible with TDNN:

• Predict process' values
• Find a relationship between two processes.

Some RNN, like NARX also allow you to do that, and it is also used to predict financial time series, usually better than TDNN.

A TDNN looks more like a feedforward network, because time aspect is only inserted through its inputs, unlike NARX that also needs the predicted/real future value as input. This characteristic makes TDNN less robust than NARX for predicting values, but requires less processing and is easier to train.

If you are trying to find a relationship between a process $X(t)$ and a process $Y(t)$, NARX requires you to have past values of $Y$, while TDNN does not.

I recommend reading Simon Haykin's Neural Networks: A Comprehensive Foundation (2nd Edition) and this FAQ. There are lots of neural networks architectures and variations. Sometimes they have many names or there is no consensus about their classification.

Answer 2

TDNNs is a simple way to represent a mapping between past and present values. The delays in the TDNN remain constant throughout the training procedure and are estimated before by using trial and error along with some heuristics. It, however, may be the case that these fixed delays do not capture the actual temporal locations of the time dependencies. On the other hand, the "memory" feature of the RNN structures can capture this information by learning these dependencies. The problem with RNNs is that they are impractical to use when trained with traditional techniques (e.g Backpropagation through time) for learning long term dependencies. This problem arises from the so-called "vanishing/exploding" gradient which basically means that as we propagate the error signals backwards through the networks structure they tend to vanish or explode. More advanced recurrent structures (eg LSTM) have properties that mitigate this problem and can learn long term dependencies and are particularly suited for learning sequential data.