When should RNNs be used, seeing as they take much longer to train?

Question

RNNs seem to take much longer to train in most if not all cases. I assume this is because the number of operations involved in training an RNN scales not only with the number of examples being fed into the network, but also with the number of timestamps. Where a regular network does one operation, an RNN does $n$, where $n$ is the number of timestamps per sequence.

For various text and time series related problems, I've switched out RNN layers (i.e. LSTM or GRU layers) with pooling layers that simply average over the temporal dimension. This leaves you with an ordinary network with Dense layers. These non-recurrent networks have always performed just as well as the RNN, but they train much faster.

Which brings me to my question. RNNs require more training time to reach the same level of performance as a non-recurrent network. So what is the point of using an RNN? In what situations are RNNs are a better choice than averaging over the time dimension and using a run-of-the-mill Dense network?

Answer 1

In some of the problems your model should have memory and ability to recall objects. For example in image captioning, your model should learn to generate some words based on the input image. You can not use just CNN or MLPs to solve it because this models only can extract some features from the image and they can not remember what word should be generated after seeing a tree as input. RNNs solve this problem by using feedbacks(as you know, feedbacks are the memory units in mathematical problems). So, when your informations are stored in a sequence words, video frames, and ...) you need to use RNN. You can read this article for more information.