16
$\begingroup$

Why do recurrent neural networks (RNNs) have a tendency to suffer from vanishing/exploding gradient?

For what a vanishing/exploding gradient is, see Pascanu, et al. (2013). On the difficulty of training recurrent neural networks, section 2 (pdf).

$\endgroup$
3
  • 2
    $\begingroup$ You will need to copy whatever context is necessary to understand your question into the body of the question itself. People aren't going to want to read the pdf to answer your question. $\endgroup$ Mar 6, 2015 at 3:27
  • 2
    $\begingroup$ @gung I shouldn't have to give any context because vanishing/exploding gradient problem is well-known problem in deep learning, especially with recurrent neural networks. In other words, it is basic knowledge that (vanilla versions of) RNN's suffer from the vanishing/exploding gradient problem. The Why is not basic knowledge. $\endgroup$
    – user70394
    Mar 6, 2015 at 3:45
  • $\begingroup$ a possible approach that works for deep NNs is to have 'pass through' connections. arxiv.org/abs/1512.03385. maybe this works for RNN too. $\endgroup$
    – seanv507
    Jan 30, 2016 at 13:10

4 Answers 4

14
$\begingroup$

TL;DR

The main reasons are the following traits of BPTT:

  1. An unrolled RNN tends to be a very deep network.
  2. In an unrolled RNN the gradient in an early layer is a product that (also) contains many instances of the same term.

Long Version

To train an RNN, people usually use backpropagation through time (BPTT), which means that you choose a number of time steps $N$, and unroll your network so that it becomes a feedforward network made of $N$ duplicates of the original network, while each of them represents the original network in another time step.

unrolling an RNN
(image source: wikipedia)

So BPTT is just unrolling your RNN, and then using backpropagation to calculate the gradient (as one would do to train a normal feedforward network).

Cause 1: The unrolled network is usually very deep

Because our feedforward network was created by unrolling, it is $N$ times as deep as the original RNN. Thus the unrolled network is often very deep.

In deep feedforward neural networks, backpropagation has "the unstable gradient problem", as Michael Nielsen explains in the chapter Why are deep neural networks hard to train? (in his book Neural Networks and Deep Learning):

[...] the gradient in early layers is the product of terms from all the later layers. When there are many layers, that's an intrinsically unstable situation. The only way all layers can learn at close to the same speed is if all those products of terms come close to balancing out.

I.e. the earlier the layer, the longer the product becomes, and the more unstable the gradient becomes. (For a more rigorous explanation, see this answer.)

Cause 2: The product that gives the gradient contains many instances of the same term

The product that gives the gradient includes the weights of every later layer.
So in a normal feedforward neural network, this product for the $d^{\text{th}}$-to-last layer might look like: $$w_1\cdot\alpha_{1}\cdot w_2\cdot\alpha_{2}\cdot\ \cdots\ \cdot w_d\cdot\alpha_{d}$$ Nielsen explains that (with regard to absolute value) this product tends to be either very big or very small (for a large $d$).

But in an unrolled RNN, this product would look like: $$w\cdot\alpha_{1}\cdot w\cdot\alpha_{2}\cdot\ \cdots\ \cdot w\cdot\alpha_{d}$$ as the unrolled network is composed of duplicates of the same network.

Whether we are dealing with numbers or matrices, the appearance of the same term $d$ times means that the product is much more unstable (as the chances are much smaller that "all those products of terms come close to balancing out").

And so the product (with regard to absolute value) tends to be either exponentially small or exponentially big (for a large $d$).

In other words, the fact that the unrolled RNN is composed of duplicates of the same network makes the unrolled network's "unstable gradient problem" more severe than in a normal deep feedforward network.

$\endgroup$
1
  • 1
    $\begingroup$ It should be noted that cause 1 was already mentioned in Denis' answer and udani's answer (which also linked to the same chapter of Nielsen's book that I linked to), and cause 2 was already mentioned in Aaron's comment. $\endgroup$ Oct 8, 2018 at 9:54
9
$\begingroup$

Because RNN is trained by backpropagation through time, and therefore unfolded into feed forward net with multiple layers. When gradient is passed back through many time steps, it tends to grow or vanish, same way as it happens in deep feedforward nets

$\endgroup$
5
  • 1
    $\begingroup$ Let me ask a slightly different question: I thought for deep networks the problem was vanishing gradients. Why do RNN's also have the (possible) additional problem of exploding gradients? $\endgroup$
    – user70394
    Mar 6, 2015 at 15:58
  • 3
    $\begingroup$ @user70394 Sufficiently deep feed forward net can suffer from exploding gradients too. In RNNs exploding gradients happen when trying to learn long-time dependencies, because retaining information for long time requires oscillator regimes and these are prone to exploding gradients. See this paper for RNN specific rigorous mathematical discussion of the problem. $\endgroup$ Mar 6, 2015 at 16:20
  • $\begingroup$ Does deep feedforward nets suffer from gradient exploding? $\endgroup$
    – n0p
    Sep 16, 2016 at 7:02
  • 1
    $\begingroup$ In an RNN you are repeatedly multiplying by the same weight matrix. This is what makes the exploding gradient problem worse than it is in deep feed forward net. $\endgroup$
    – Aaron
    Apr 22, 2018 at 22:19
  • $\begingroup$ @DenisTarasov You have linked to an older version of the paper that the OP already linked to. This is the up to date version (and the one the OP linked to). $\endgroup$ Oct 5, 2018 at 17:20
1
$\begingroup$

I would like to point out one point that the answers above seems to have missed about vanishing gradient in RNN.

What people mean by vanishing gradient should be understood differently from the original meaning in DNN. But first we need to make some notation.

Let $h_0 \neq 0$, the recursive formula for Elman Recurrent Neural Network is \begin{align*} h_t &= f_h(U_hx_t + W_hh_{t-1} + b_h) \\ \hat y_t &= f_y(W_y h_t +b_y) \end{align*} For $1\leq t \leq T$, as $T$ is the total of time steps.

Denote $E_t$ as the error between real value $y_t$ and $\hat y_t$, then the total loss is $L = \sum_{i=1}^T E_t$. Due to shared weight nature of RNN, finding partial derivative of $L$ w.r.t to $W_{hh}$ obliges you to find $\frac{\partial E_t}{\partial W}$ for each $W$ w.r.t each time-stamp $i<t$.

Then if you look at the paper where most of what our current understand about the exploding/vanishing gradient is based upon:

This term [$\frac{\partial E_t}{\partial W}$ at $i$] tends to become very small in comparison to terms for which $\tau$ is close to $t$. This means that even though there might exist a change in $W$ that would allow a, to jump to another (better) basin of attraction, the gradient of the cost with respect to $W$ does not reflect that possibility.

What this means when $||W||$ is small, some partial derivative at time-stamp $i$ of some component $E_t$ might get lost due to their time distance. Resulting in a gradient descent algorithm that pays too much attention to the surrounding (usually bumpy) loss surface that not necessarily go down in the long run.

So what people usually mean by vanishing gradient in RNN is only by long component that contain distance information of RNN, not the system as a whole.

$\endgroup$
0
$\begingroup$

This chapter describes the reason for vanishing gradient problem really well. When we unfold the RNN over time it is also like a deep neural network. Therefore according to my understanding it also suffers from vanishing gradient problem as deep feedforward nets.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.