Recurrent neural networks - why is the vanishing gradient a problem?

Question

I understand why the gradient tends to become small in the early layers of a deep network. However I am trying to clarify why this is a problem in the case of RNNs.

This is what I understand: the gradient $\frac{\partial E}{\partial w} $ of a weight $w$ is computed as the average of the gradients at each time step. If the gradients of the early time steps are close to $0$, then they have no influence on the final gradient. Effectively this means the beginning of the sequence has no influence on how the RNN weights are learnt.

Is this the problematic aspect of the vanishing gradient?

Answer 1

If I understood your question properly, then what you have thought of vanishing gradient in RNN is wrong.

In RNN we known compute the current timestep value based on previous output values .

The main reason we compute gradients of the weights is to see how much of influences does it make on the network's output. In RNN we take in previous output to compute current value.

The above image is from wildml where Denny Britz has written elegantly.

The red lines in the image is where we calculate the gradients. So while we compute chain rule to find the gradients, if the previous time step computation we take keeps on increasing then the gradient values will get small. This is known as Vanishing Gradient and a problem in RNN. We use LSTM, GRU to solve these problems.