In the context of RNNs, gradient vanishing refers to the fact the gradient signal decays to zero as we approach the beginning of the sequence during the unfolding of the network in backpropagation through time. The equations are such that the signal for early layers is multiplied by many fractions smaller than 1, hence it tends to approach zero.
My question may seem naïve, but assuming the backpropagation algorithm calculates the exact gradients of the loss with respect to the parameters, why is gradient vanishing a problem (apart from the issue of numerical stability)?
If the gradients backprop calculates are such that earlier time steps have small gradient, doesn't this just reflect the relative importance of these time steps on the decision? Otherwise, these time steps would have had larger gradients (again - assuming backprop calculates exact gradients).