Recurrent vs Recursive Neural Networks: Which is better for NLP?

Question

There are Recurrent Neural Networks and Recursive Neural Networks. Both are usually denoted by the same acronym: RNN. According to Wikipedia, Recurrent NN are in fact Recursive NN, but I don't really understand the explanation.

Moreover, I don't seem to find which is better (with examples or so) for Natural Language Processing. The fact is that, although Socher uses Recursive NN for NLP in his tutorial, I can't find a good implementation of recursive neural networks, and when I search in Google, most of the answers are about Recurrent NN.

Besides that, is there another DNN which applies better for NLP, or it depends on the NLP task? Deep Belief Nets or Stacked Autoencoders? (I don't seem to find any particular util for ConvNets in NLP, and most of the implementations are with machine vision in mind).

Finally, I would really prefer DNN implementations for C++ (better yet if it has GPU support) or Scala (better if it has Spark support) rather than Python or Matlab/Octave.

I've tried Deeplearning4j, but it's under constant development and the documentation is a little outdated and I can't seem to make it work. Too bad because it has the "black box" like way of doing things, very much like scikit-learn or Weka, which is what I really want.

Answer 1

Recurrent Neural networks are recurring over time. For example if you have a sequence

x = ['h', 'e', 'l', 'l']

This sequence is fed to a single neuron which has a single connection to itself.

At time step 0, the letter 'h' is given as input.At time step 1, 'e' is given as input. The network when unfolded over time will look like this.

A recursive network is just a generalization of a recurrent network. In a recurrent network the weights are shared (and dimensionality remains constant) along the length of the sequence because how would you deal with position-dependent weights when you encounter a sequence at test-time of different length to any you saw at train-time. In a recursive network the weights are shared (and dimensionality remains constant) at every node for the same reason.

This means that all the W_xh weights will be equal(shared) and so will be the W_hh weight. This is simply because it is a single neuron which has been unfolded in time.

This is what a Recursive Neural Network looks like.

It is quite simple to see why it is called a Recursive Neural Network. Each parent node's children are simply a node similar to that node.

The Neural network you want to use depends on your usage. In Karpathy's blog, he is generating characters one at a time so a recurrent neural network is good.

But if you want to generate a parse tree, then using a Recursive Neural Network is better because it helps to create better hierarchical representations.

If you want to do deep learning in c++, then use CUDA. It has a nice user-base, and is fast. I do not know more about that so cannot comment more.

In python, Theano is the best option because it provides automatic differentiation, which means that when you are forming big, awkward NNs, you don't have to find gradients by hand. Theano does it automatically for you. This feature is lacked by Torch7.

Theano is very fast as it provides C wrappers to python code and can be implemented on GPUs. It also has an awesome user base, which is very important while learning something new.

Answer 2

Large Recurrent Neural Networks are considered maybe the most powerful model for NLP. A great article written by A. Karpathy on Recurrent Neural Networks and character level modeling is available at http://karpathy.github.io/2015/05/21/rnn-effectiveness/

Having tried a large number of libraries for deep learning (theano, caffe etc.). I would strongly suggest the use Torch7 which is considered the state-of-the-art tool for NNs and it supported by NYU, Facebook AI and Google DeepMind. Torch7 is based on lua and there are so many examples that you can easily familiarize with. A lot of code can be found on github, a good start would be https://github.com/wojzaremba/lstm.

Finally, the beauty of lua is that LuaJIT can be injected very easily in Java, Python, Matlab etc.

Answer 3

Recurrent Neural Networks (RNN) basically unfolds over time. It is used for sequential inputs where the time factor is the main differentiating factor between the elements of the sequence. For example, here is a recurrent neural network used for language modeling that has been unfolded over time. At each time step, in addition to the user input at that time step, it also accepts the output of the hidden layer that was computed at the previous time step.

---

A Recursive Neural Networks is more like a hierarchical network where there is really no time aspect to the input sequence but the input has to be processed hierarchically in a tree fashion. Here is an example of how a recursive neural network looks. It shows the way to learn a parse tree of a sentence by recursively taking the output of the operation performed on a smaller chunk of the text.

---

[NOTE]:

LSTM and GRU are two extended RNNs types with the forget gate, which are highly common in NLP.

---

LSTM-Cell Formula:

Answer 4

To answer a couple of the questions:

CNNs definitely are used for NLP tasks sometimes. They are one way to take a variable-length natural language input and reduce it to a fixed length output such as a sentence embedding. Google's Multilingual Universal Sentence Encoder (USE) is one example:

• https://arxiv.org/abs/1907.04307
• https://tfhub.dev/google/universal-sentence-encoder-multilingual/3

Since this question has been asked, there have been a number of new models proposed for NLP that are distinct from those mentioned above such as Transformers and pre-trained neural language models like BERT and some of the other flavors of USE. https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)