In the context of RNNs, gradient vanishing refers to the fact the gradient signal decays to zero as we approach the beginning of the sequence during the unfolding of the network in backpropagation through time. The equations are such that the signal for early layers is multiplied by many fractions smaller than 1, hence it tends to approach zero.

My question may seem naïve, but assuming the backpropagation algorithm calculates the exact gradients of the loss with respect to the parameters, why is gradient vanishing a problem (apart from the issue of numerical stability)?
If the gradients backprop calculates are such that earlier time steps have small gradient, doesn't this just reflect the relative importance of these time steps on the decision? Otherwise, these time steps would have had larger gradients (again - assuming backprop calculates exact gradients).


1 Answer 1


The problem is that the gradient vanishes exponentially fast w.r.t. depth. That means even moderately deep networks may get paralyzed by this, i.e. all the layers except the last few are not learning or learning incredibly slow. Such network is then using only a small ratio of its weights, the rest is just random noise produced by the initialization.

The fact that layers getting a tiny gradient did not contribute much to the error does not mean they have optimal values and don't need to train.

  • $\begingroup$ Thanks for you answer. Is it right to say, then, that the problem stems from the fact the loss surface of a vanilla RNN is such that early time steps tend not to influence on the final decision (and hence have small gradients)? $\endgroup$ May 22, 2018 at 16:00
  • $\begingroup$ I think that is something different $\endgroup$ May 24, 2018 at 12:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.