4
$\begingroup$

I am doing a research about NLP and I am using RNN (Recurrent Neural Network) or CNN (Convolutional Neural Network) to encode a sentence into a vector. When using CNN, the training time is significantly smaller than RNN. It is natural to me to think that CNN is faster than RNN because it does not build the relationship between hidden vectors of each timesteps, so it takes less time to feed forward and back propagate. However, this question was asked recently by another person and I do not have any evidence to support my intuition.

Is there any evidence or results about comparing the speed of CNN and RNN, especially on NLP tasks?

$\endgroup$
16
$\begingroup$

I have done some projects on text classification and relation extraction using CNN and RNN (specifically, LSTM and GRU): CNNs tend to be much faster (~5 times faster) than RNN.

It's hard to draw fair comparisons:

  • CNN and RNN have different hyperparameters (filter dimension, number of filters, hidden state dimension, etc.)
  • there exist many sort of RNNs
  • the running time depends on the implementation, especially RNNs.
  • CNNs run faster with CuDNN + CNMeM. RNNs benefit less from them.
  • etc.

Nvidia has historically focused much more on CNN than RNN, as computer vision mostly employs CNN.

One benchmark: https://github.com/baidu-research/DeepBench

enter image description here

FYI:

$\endgroup$
  • $\begingroup$ Thank you very much for your answer. It is very helpful. $\endgroup$ – The Lazy Log Feb 17 '17 at 6:46
1
$\begingroup$

As already said by Franck Dernoncourt, the answer depends on your model. CNN and RNN are different architectures, used differently, usually for different purposes. You can't really replace one by another without changing other elements of the model to compare the performance.

However, CNN's are faster by design, since the computations in CNN's can happen in parallel (same filter applied to multiple locations of the image at the same time), while RNN's need to be processed sequentially, since the subsequent steps depend on previous ones. This was the reason why Bradbury et al (2016) introduced the Quasi-Recurrent Neural Networks that use some of the CNN components to imitate RNN's while speeding them up.

$\endgroup$
  • $\begingroup$ I’ve always had a bit of trouble understanding the parallelization issue with RNNs. I get that you have to apply them once at each time step (in tandem). But I would have figured you can easily parallelize the inner matrix multiplications within a time step, especially if the hidden layer is large. Given how simple those are I’m surprised the RNNs end up slower. $\endgroup$ – Alex R. Sep 28 '18 at 6:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.