When to use recurrent neural networks?

Question

In order to create a training set for RNNs one typically takes a sequence and turns it into a training set using the sliding window approach. For example if the sequence is:

1 2 3 4 5 6 7 8 9 

One can form a training set using a window size of 4 as follows:

1 2 3 4
2 3 4 5
3 4 5 6
4 5 6 7
5 6 7 8
6 7 8 9

In the above training set the first 3 columns are input features and the last column is the output (in a more realistic setting larger window sizes can be used).

Now actually this is a classical supervised machine learning training set. So I can use any supervised regression model such as linear regression or multilayer NNs. I can use an RNN model too. My question is this: what properties of the data sequence make RNNs a better predictor than linear regression or multilayer NNs? I anticipate some kind of answer like: when the past data is important for prediction. However, the past data (i.e., input features) is available for all methods. There must be some kind of structure in the inputs that makes RNNs suitable for prediction. If you can also also give some simple example sequences for illustration, it would be great.

Thanks

Answer 1

Firstly, while you're specifically asking about future prediction, RNNs can be used in a variety of applications, including translation from one language to another, detecting a music instrument player in an audio file, speech recognition, and others.

RNNs are especially well setup to deal with time-dependent series, since you are feeding it your input sequence step-by-step, one value at at time, and each value gets incorporated into the RNN in the form of "memory" (which can be short- or long-term). So, by design, not all points in your input series are treated equally, RNN "knows" about the order. At each step RNN can decide what information from the past to drop or pass on to the next step.

Another advantage of RNNs is that you set it up such that you can feed sequences or variable length to it and it will still perform well. It can also output variable length sequences. This is especially useful in applications like speech recognition, translation, sentiment analysis, and others where the input and/or output cannot be made to be constant length.

For many applications, including the one you described in your question, you can indeed use other ML algorithms and they may work perfectly well for you. But it's the versatility of RNNs combined with the power of deep learning that makes them so useful for processing ordered sequences and it's the reason why it's used in most of the state-of-the-art NLP applications.