Recurrent Neural Network vs Traditional Neural Network

Question

I've read a lot about the difference between RNN and traditional NNs (MPLs) but I have still doubts. Below I would like to ask a specific question.

Assume that we have a sequence data: $s_1, s_2, s_3, ..., s_n.$ My aim is to make a prediction for $s_i$ given $k$ previous data points. Assume that $k=2$ for this example. I create the following data set:

$$ s_1, s_2, s_3 \\ s_2, s_3, s_4 \\ s_3, s_4, s_5 \\ ... \\ s_{n-2}, s_{n-1}, s_n \\ $$

The third column is $y$, that is, the target to predict. Now, I can use this dataset to train a MLP or a RNN (LSTM or similar). After they are trained, given two consecutive data points I can use either models (MLP or RNN) to predict the next data point.

My question is, what is the advantage of using a RNN instead of a MLP for this problem? If possible I would be glad if somebody gives a simple example sequence to illustrate the advantage of using RNN over MLP. Thanks.

Answer 1

First of all, with RNN you need much less parameters, since you condition only on the current input and the hidden state returned by previous RNN cell. You only need parameters for those signals, not for the all the inputs. This should be obvious if you think of how RNN cells are defined. Quoting myself

As described by Andrey Karpathy, the basic recurrent neural network cell is something like

$$ h_t = \tanh(W_{hh}h_{t-1} + W_{xh}x_t) $$

so it takes previous hidden state $h_{t-1}$ and current input $x_t$, to produce hidden state $h_t$. Notice that $W_{hh}$ and $W_{xh}$ are not indexed by time $t$, we use the same weights for each timestep. In simplified python code, the forward pass is basically a for-loop:

for t in range(timesteps):
    h[t] = np.tanh(np.dot(Wxh, x[t]) + np.dot(Whh, h[t-1]))

Second, because of the above fact, RNNs can take as inputs sequences of arbitrary length. This is impossible for classical architectures using only the dense layers, since they need fixed size of inputs. Notice however, that nowadays we also have other neural network architectures that can deal with inputs of arbitrary length, that don't use RNNs, for example Transformer network (Vaswani et al, 2017).