Is Convolutional Neural Network (CNN) faster than Recurrent Neural Network (RNN)?

Question

I am doing a research about NLP and I am using RNN (Recurrent Neural Network) or CNN (Convolutional Neural Network) to encode a sentence into a vector. When using CNN, the training time is significantly smaller than RNN. It is natural to me to think that CNN is faster than RNN because it does not build the relationship between hidden vectors of each timesteps, so it takes less time to feed forward and back propagate. However, this question was asked recently by another person and I do not have any evidence to support my intuition.

Is there any evidence or results about comparing the speed of CNN and RNN, especially on NLP tasks?

Answer 1

I have done some projects on text classification and relation extraction using CNN and RNN (specifically, LSTM and GRU): CNNs tend to be much faster (~5 times faster) than RNN.

It's hard to draw fair comparisons:

• CNN and RNN have different hyperparameters (filter dimension, number of filters, hidden state dimension, etc.)
• there exist many sort of RNNs
• the running time depends on the implementation, especially RNNs.
• CNNs run faster with CuDNN + CNMeM. RNNs benefit less from them.
• etc.

Nvidia has historically focused much more on CNN than RNN, as computer vision mostly employs CNN.

One benchmark: https://github.com/baidu-research/DeepBench

FYI:

• Benchmarks based on neural networks libraries to compare the performance between different GPUs
• Why doesn't training RNNs use 100% of the GPU?

Answer 2

As already said by Franck Dernoncourt, the answer depends on your model. CNN and RNN are different architectures, used differently, usually for different purposes. You can't really replace one by another without changing other elements of the model to compare the performance.

However, CNN's are faster by design, since the computations in CNN's can happen in parallel (same filter applied to multiple locations of the image at the same time), while RNN's need to be processed sequentially, since the subsequent steps depend on previous ones. This was the reason why Bradbury et al (2016) introduced the Quasi-Recurrent Neural Networks that use some of the CNN components to imitate RNN's while speeding them up.