Why do RNNs have a tendency to suffer from vanishing/exploding gradient?

Question

Why do recurrent neural networks (RNNs) have a tendency to suffer from vanishing/exploding gradient?

For what a vanishing/exploding gradient is, see Pascanu, et al. (2013). On the difficulty of training recurrent neural networks, section 2 (pdf).

Answer 1

TL;DR

The main reasons are the following traits of BPTT:

1. An unrolled RNN tends to be a very deep network.
2. In an unrolled RNN the gradient in an early layer is a product that (also) contains many instances of the same term.

Long Version

To train an RNN, people usually use backpropagation through time (BPTT), which means that you choose a number of time steps $N$, and unroll your network so that it becomes a feedforward network made of $N$ duplicates of the original network, while each of them represents the original network in another time step.

(image source: wikipedia)

So BPTT is just unrolling your RNN, and then using backpropagation to calculate the gradient (as one would do to train a normal feedforward network).

Cause 1: The unrolled network is usually very deep

Because our feedforward network was created by unrolling, it is $N$ times as deep as the original RNN. Thus the unrolled network is often very deep.

In deep feedforward neural networks, backpropagation has "the unstable gradient problem", as Michael Nielsen explains in the chapter Why are deep neural networks hard to train? (in his book Neural Networks and Deep Learning):

[...] the gradient in early layers is the product of terms from all the later layers. When there are many layers, that's an intrinsically unstable situation. The only way all layers can learn at close to the same speed is if all those products of terms come close to balancing out.

I.e. the earlier the layer, the longer the product becomes, and the more unstable the gradient becomes. (For a more rigorous explanation, see this answer.)

Cause 2: The product that gives the gradient contains many instances of the same term

The product that gives the gradient includes the weights of every later layer.
So in a normal feedforward neural network, this product for the $d^{\text{th}}$-to-last layer might look like: $$w_1\cdot\alpha_{1}\cdot w_2\cdot\alpha_{2}\cdot\ \cdots\ \cdot w_d\cdot\alpha_{d}$$ Nielsen explains that (with regard to absolute value) this product tends to be either very big or very small (for a large $d$).

But in an unrolled RNN, this product would look like: $$w\cdot\alpha_{1}\cdot w\cdot\alpha_{2}\cdot\ \cdots\ \cdot w\cdot\alpha_{d}$$ as the unrolled network is composed of duplicates of the same network.

Whether we are dealing with numbers or matrices, the appearance of the same term $d$ times means that the product is much more unstable (as the chances are much smaller that "all those products of terms come close to balancing out").

And so the product (with regard to absolute value) tends to be either exponentially small or exponentially big (for a large $d$).

In other words, the fact that the unrolled RNN is composed of duplicates of the same network makes the unrolled network's "unstable gradient problem" more severe than in a normal deep feedforward network.

Answer 2

Because RNN is trained by backpropagation through time, and therefore unfolded into feed forward net with multiple layers. When gradient is passed back through many time steps, it tends to grow or vanish, same way as it happens in deep feedforward nets

Answer 3

I would like to point out one point that the answers above seems to have missed about vanishing gradient in RNN.

What people mean by vanishing gradient should be understood differently from the original meaning in DNN. But first we need to make some notation.

Let $h_0 \neq 0$, the recursive formula for Elman Recurrent Neural Network is \begin{align*} h_t &= f_h(U_hx_t + W_hh_{t-1} + b_h) \\ \hat y_t &= f_y(W_y h_t +b_y) \end{align*} For $1\leq t \leq T$, as $T$ is the total of time steps.

Denote $E_t$ as the error between real value $y_t$ and $\hat y_t$, then the total loss is $L = \sum_{i=1}^T E_t$. Due to shared weight nature of RNN, finding partial derivative of $L$ w.r.t to $W_{hh}$ obliges you to find $\frac{\partial E_t}{\partial W}$ for each $W$ w.r.t each time-stamp $i<t$.

Then if you look at the paper where most of what our current understand about the exploding/vanishing gradient is based upon:

This term [$\frac{\partial E_t}{\partial W}$ at $i$] tends to become very small in comparison to terms for which $\tau$ is close to $t$. This means that even though there might exist a change in $W$ that would allow a, to jump to another (better) basin of attraction, the gradient of the cost with respect to $W$ does not reflect that possibility.

What this means when $||W||$ is small, some partial derivative at time-stamp $i$ of some component $E_t$ might get lost due to their time distance. Resulting in a gradient descent algorithm that pays too much attention to the surrounding (usually bumpy) loss surface that not necessarily go down in the long run.

So what people usually mean by vanishing gradient in RNN is only by long component that contain distance information of RNN, not the system as a whole.

Answer 4

This chapter describes the reason for vanishing gradient problem really well. When we unfold the RNN over time it is also like a deep neural network. Therefore according to my understanding it also suffers from vanishing gradient problem as deep feedforward nets.