I have seen the word "vanishing gradient" many times in deep learning literature. what is that? gradient respect to what variable? input variable or hidden units?

Does that mean the gradient vector is all zero? Or the optimization stuck in local minima / saddle point?

  • $\begingroup$ You are minimizing error. It is with respect to parameters/coefficients/thetas/weights. $\endgroup$
    – ksha
    Commented Sep 4, 2017 at 7:15
  • $\begingroup$ @ketan all parameters or a subset of parameters in the middle layer? $\endgroup$
    – Haitao Du
    Commented Sep 4, 2017 at 7:17
  • $\begingroup$ for a neural network, for example, ideally backpropagation should update all the weights right from final layer's weight matrix to the first weight matrix, the one that multiplies with input vector to give the first hidden unit. $\endgroup$
    – ksha
    Commented Sep 4, 2017 at 7:20
  • 2
    $\begingroup$ These notes from Hinton's course explain this problem: cs.toronto.edu/~hinton/csc2535/notes/lec10new.pdf $\endgroup$ Commented Sep 4, 2017 at 7:38
  • $\begingroup$ thanks for the link! @JakubBartczuk This is really helpful for me since it also talked both HMM and RNN! See my question here $\endgroup$
    – Haitao Du
    Commented Sep 4, 2017 at 18:00

3 Answers 3


If you do not carefully choose the range of the initial values for the weights, and if you do not control the range of the values of the weights during training, vanishing gradient would occur which is the main barrier to learning deep networks. The neural networks are trained using the gradient descent algorithm: $$w^{new} := w^{old} - \eta \frac{\partial L}{\partial w}$$ where $L$ is the loss of the network on the current training batch. It is clear that if the $\frac{\partial L}{\partial w}$ is very small, the learning will be very slow, since the changes in $w$ will be very small. So, if the gradients are vanished, the learning will be very very slow.

The reason for vanishing gradient is that during backpropagation, the gradient of early layers (layers near to the input layer) are obtained by multiplying the gradients of later layers (layers near to the output layer). So, for example if the gradients of later layers are less than one, their multiplication vanishes very fast.

With this explanations these are answers to your questions:

  • Gradient is the gradient of the loss with respect to each trainable parameters (weights and biases).
  • Vanishing gradient does not mean the gradient vector is all zero (except for numerical underflow), but it means the gradients are so small that the learning will be very slow.
  • $\begingroup$ Vanishing gradient does not mean the gradient vector is all zero could you explain more detail? I don't understand that in the gradient decent, we do not want to achieve the exact zero gradient and will stop when the parameter unchanges within a $\epsilon,$ which is the same case of vanishing gradient. $\endgroup$ Commented Sep 30, 2020 at 5:34
  • $\begingroup$ You do describe a vanishing gradient, however at this point in training this is also called convergence as well. I.E. where the network converges to its global (or local) optimum. If this location is matched with a strong testing performance then our vanishing gradient is not a problem. If anything it helps assure us that our model has finished training and we can stop $\endgroup$
    – Greg
    Commented Jan 24, 2022 at 10:09
  • $\begingroup$ I don't get why vanishing gradient affects only the earlier layer. Just consider a simple network with 5 hidden layers, 1 neuron per layer and identity activations, i.e: $y = w_5 w_4 w_3 w_2 w_1 x$. The gradient with respect to $w_1$ is simply $w_5 w_4 w_3 w_2 x$ and the gradient with respect to $w_5$ is simply $w_4 w_3 w_2 w_1 x$. Many multiplications are present in both cases, so they will vanish or explode both. $\endgroup$
    – ado sar
    Commented Feb 9 at 18:37

Consider the following feedforward neural network:

example network

  • Let $w^l_{j,k}$ be the weight for the connection from the $k^{\text{th}}$ neuron in the $(l-1)^{\text{th}}$ layer to the $j^{\text{th}}$ neuron in the $l^{\text{th}}$ layer.
  • Let $b^l_j$ be the bias of the $j^{\text{th}}$ neuron in the $l^{\text{th}}$ layer.
  • Let $C$ be the cost function. We consider the inputs and desired outputs of training examples as constants while we train our network, so in our simple network, $C$ is a function of the weights and biases in the network. (I.e. weights and biases of hidden layers and the output layer.)

  • Let $\delta^l\equiv\left(\begin{gathered}\frac{\partial C}{\partial w_{1,1}^{l}}\\ \\ \frac{\partial C}{\partial w_{1,2}^{l}}\\ \\ \frac{\partial C}{\partial w_{2,1}^{l}}\\ \\ \frac{\partial C}{\partial w_{2,2}^{l}}\\ \\ \frac{\partial C}{\partial b_{1}^{l}}\\ \\ \frac{\partial C}{\partial b_{2}^{l}} \end{gathered} \right)$ be "the gradient in the $l^{\text{th}}$ layer".

(I use the notation used by Michael Nielsen in the excellent chapter How the backpropagation algorithm works in the book Neural Networks and Deep Learning, except for "the gradient in the $l^{\text{th}}$ layer", which I define slightly differently.)

I am not aware of a strict definition of the vanishing gradient problem, but I think Nielsen's definition (from the chapter Why are deep neural networks hard to train? in the same book) is quite clear:

[...] in at least some deep neural networks, the gradient tends to get smaller as we move backward through the hidden layers. This means that neurons in the earlier layers learn much more slowly than neurons in later layers. [...] The phenomenon is known as the vanishing gradient problem.

E.g. in our network, if $||\delta^2||\ll||\delta^4||\ll||\delta^6||$, then we say we have a vanishing gradient problem.

If we use Stochastic Gradient Descent, then the size of the change to every parameter $\alpha$ (e.g. a weight, a bias, or any other parameter in more sophisticated networks) in each step taken by the algorithm (we might call this size "the speed of learning of $\alpha$") is proportional to an approximation of $-\frac{\partial C}{\partial\alpha}$ (based on a mini-batch of training examples).

Thus, in case of a vanishing gradient problem, we can say that the speed of learning of parameters of neurons becomes lower and lower, as you move to earlier layers.

So it doesn't necessarily mean that gradients in earlier layers are actually zero, or that they are stuck in any manner, but their speed of learning is low enough to significantly increase the training time, which is why it is called "vanishing gradient problem".

See this answer for a more rigorous explanation of the problem.


Continuing from comments, when you use sigmoid activation function which squashes the input to a small range $(0,1)$, you further multiply it by a small learning rate and more partial derivatives (chain rule) as you go back in layers. The value of delta to be updated diminishes and thus earlier layers get little or no updates. If little, then it would require lot of training. If no, then only changing the activation function (AF) would be of any help. RELUs are currently the best AFs that avoid this problem.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.