The nature of the problem of vanishing gradients in RNN

Question

In the context of RNNs, gradient vanishing refers to the fact the gradient signal decays to zero as we approach the beginning of the sequence during the unfolding of the network in backpropagation through time. The equations are such that the signal for early layers is multiplied by many fractions smaller than 1, hence it tends to approach zero.

My question may seem naïve, but assuming the backpropagation algorithm calculates the exact gradients of the loss with respect to the parameters, why is gradient vanishing a problem (apart from the issue of numerical stability)?
If the gradients backprop calculates are such that earlier time steps have small gradient, doesn't this just reflect the relative importance of these time steps on the decision? Otherwise, these time steps would have had larger gradients (again - assuming backprop calculates exact gradients).

Answer 1

The problem is that the gradient vanishes exponentially fast w.r.t. depth. That means even moderately deep networks may get paralyzed by this, i.e. all the layers except the last few are not learning or learning incredibly slow. Such network is then using only a small ratio of its weights, the rest is just random noise produced by the initialization.

The fact that layers getting a tiny gradient did not contribute much to the error does not mean they have optimal values and don't need to train.