Why do recurrent neural networks (RNNs) have a tendency to suffer from vanishing/exploding gradient?

For what a vanishing/exploding gradient is, see Pascanu, et al. (2013). On the difficulty of training recurrent neural networks, section 2 (pdf).

  • 2
    $\begingroup$ You will need to copy whatever context is necessary to understand your question into the body of the question itself. People aren't going to want to read the pdf to answer your question. $\endgroup$ – gung - Reinstate Monica Mar 6 '15 at 3:27
  • 1
    $\begingroup$ @gung I shouldn't have to give any context because vanishing/exploding gradient problem is well-known problem in deep learning, especially with recurrent neural networks. In other words, it is basic knowledge that (vanilla versions of) RNN's suffer from the vanishing/exploding gradient problem. The Why is not basic knowledge. $\endgroup$ – user70394 Mar 6 '15 at 3:45
  • $\begingroup$ a possible approach that works for deep NNs is to have 'pass through' connections. arxiv.org/abs/1512.03385. maybe this works for RNN too. $\endgroup$ – seanv507 Jan 30 '16 at 13:10

Because RNN is trained by backpropagation through time, and therefore unfolded into feed forward net with multiple layers. When gradient is passed back through many time steps, it tends to grow or vanish, same way as it happens in deep feedforward nets

  • 1
    $\begingroup$ Let me ask a slightly different question: I thought for deep networks the problem was vanishing gradients. Why do RNN's also have the (possible) additional problem of exploding gradients? $\endgroup$ – user70394 Mar 6 '15 at 15:58
  • 3
    $\begingroup$ @user70394 Sufficiently deep feed forward net can suffer from exploding gradients too. In RNNs exploding gradients happen when trying to learn long-time dependencies, because retaining information for long time requires oscillator regimes and these are prone to exploding gradients. See this paper for RNN specific rigorous mathematical discussion of the problem. $\endgroup$ – Denis Tarasov Mar 6 '15 at 16:20
  • $\begingroup$ Does deep feedforward nets suffer from gradient exploding? $\endgroup$ – n0p Sep 16 '16 at 7:02
  • 1
    $\begingroup$ In an RNN you are repeatedly multiplying by the same weight matrix. This is what makes the exploding gradient problem worse than it is in deep feed forward net. $\endgroup$ – Aaron Apr 22 '18 at 22:19
  • $\begingroup$ @DenisTarasov You have linked to an older version of the paper that the OP already linked to. This is the up to date version (and the one the OP linked to). $\endgroup$ – Oren Milman Oct 5 '18 at 17:20


The main reasons are the following traits of BPTT:

  1. An unrolled RNN tends to be a very deep network.
  2. In an unrolled RNN the gradient in an early layer is a product that (also) contains many instances of the same term.

Long Version

To train an RNN, people usually use backpropagation through time (BPTT), which means that you choose a number of time steps $N$, and unroll your network so that it becomes a feedforward network made of $N$ duplicates of the original network, while each of them represents the original network in another time step.

unrolling an RNN
(image source: wikipedia)

So BPTT is just unrolling your RNN, and then using backpropagation to calculate the gradient (as one would do to train a normal feedforward network).

Cause 1: The unrolled network is usually very deep

Because our feedforward network was created by unrolling, it is $N$ times as deep as the original RNN. Thus the unrolled network is often very deep.

In deep feedforward neural networks, backpropagation has "the unstable gradient problem", as Michael Nielsen explains in the chapter Why are deep neural networks hard to train? (in his book Neural Networks and Deep Learning):

[...] the gradient in early layers is the product of terms from all the later layers. When there are many layers, that's an intrinsically unstable situation. The only way all layers can learn at close to the same speed is if all those products of terms come close to balancing out.

I.e. the earlier the layer, the longer the product becomes, and the more unstable the gradient becomes. (For a more rigorous explanation, see this answer.)

Cause 2: The product that gives the gradient contains many instances of the same term

The product that gives the gradient includes the weights of every later layer.
So in a normal feedforward neural network, this product for the $d^{\text{th}}$-to-last layer might look like: $$w_1\cdot\alpha_{1}\cdot w_2\cdot\alpha_{2}\cdot\ \cdots\ \cdot w_d\cdot\alpha_{d}$$ Nielsen explains that (with regard to absolute value) this product tends to be either very big or very small (for a large $d$).

But in an unrolled RNN, this product would look like: $$w\cdot\alpha_{1}\cdot w\cdot\alpha_{2}\cdot\ \cdots\ \cdot w\cdot\alpha_{d}$$ as the unrolled network is composed of duplicates of the same network.

Whether we are dealing with numbers or matrices, the appearance of the same term $d$ times means that the product is much more unstable (as the chances are much smaller that "all those products of terms come close to balancing out").

And so the product (with regard to absolute value) tends to be either exponentially small or exponentially big (for a large $d$).

In other words, the fact that the unrolled RNN is composed of duplicates of the same network makes the unrolled network's "unstable gradient problem" more severe than in a normal deep feedforward network.

  • $\begingroup$ It should be noted that cause 1 was already mentioned in Denis' answer and udani's answer (which also linked to the same chapter of Nielsen's book that I linked to), and cause 2 was already mentioned in Aaron's comment. $\endgroup$ – Oren Milman Oct 8 '18 at 9:54

This chapter describes the reason for vanishing gradient problem really well. When we unfold the RNN over time it is also like a deep neural network. Therefore according to my understanding it also suffers from vanishing gradient problem as deep feedforward nets.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.