What is vanishing gradient?

Question

I have seen the word "vanishing gradient" many times in deep learning literature. What is that? Gradient respects to what variable? input variable or hidden units?

Does that mean the gradient vector is all zero? Or the optimization stuck in local minima / saddle point?

Answer 1

If you do not carefully choose the range of the initial values for the weights, and if you do not control the range of the values of the weights during training, vanishing gradient would occur which is the main barrier to learning deep networks. The neural networks are trained using the gradient descent algorithm: $$w^{new} := w^{old} - \eta \frac{\partial L}{\partial w}$$ where $L$ is the loss of the network on the current training batch. It is clear that if the $\frac{\partial L}{\partial w}$ is very small, the learning will be very slow, since the changes in $w$ will be very small. So, if the gradients are vanished, the learning will be very very slow.

The reason for vanishing gradient is that during backpropagation, the gradient of early layers (layers near to the input layer) are obtained by multiplying the gradients of later layers (layers near to the output layer). So, for example if the gradients of later layers are less than one, their multiplication vanishes very fast.

With this explanations these are answers to your questions:

• Gradient is the gradient of the loss with respect to each trainable parameters (weights and biases).
• Vanishing gradient does not mean the gradient vector is all zero (except for numerical underflow), but it means the gradients are so small that the learning will be very slow.

Answer 2

Consider the following feedforward neural network:

• Let $w^l_{j,k}$ be the weight for the connection from the $k^{\text{th}}$ neuron in the $(l-1)^{\text{th}}$ layer to the $j^{\text{th}}$ neuron in the $l^{\text{th}}$ layer.
• Let $b^l_j$ be the bias of the $j^{\text{th}}$ neuron in the $l^{\text{th}}$ layer.
• Let $C$ be the cost function. We consider the inputs and desired outputs of training examples as constants while we train our network, so in our simple network, $C$ is a function of the weights and biases in the network. (I.e. weights and biases of hidden layers and the output layer.)
  
  
• Let $\delta^l\equiv\left(\begin{gathered}\frac{\partial C}{\partial w_{1,1}^{l}}\\ \\ \frac{\partial C}{\partial w_{1,2}^{l}}\\ \\ \frac{\partial C}{\partial w_{2,1}^{l}}\\ \\ \frac{\partial C}{\partial w_{2,2}^{l}}\\ \\ \frac{\partial C}{\partial b_{1}^{l}}\\ \\ \frac{\partial C}{\partial b_{2}^{l}} \end{gathered} \right)$ be "the gradient in the $l^{\text{th}}$ layer".

(I use the notation used by Michael Nielsen in the excellent chapter How the backpropagation algorithm works in the book Neural Networks and Deep Learning, except for "the gradient in the $l^{\text{th}}$ layer", which I define slightly differently.)

I am not aware of a strict definition of the vanishing gradient problem, but I think Nielsen's definition (from the chapter Why are deep neural networks hard to train? in the same book) is quite clear:

[...] in at least some deep neural networks, the gradient tends to get smaller as we move backward through the hidden layers. This means that neurons in the earlier layers learn much more slowly than neurons in later layers. [...] The phenomenon is known as the vanishing gradient problem.

E.g. in our network, if $||\delta^2||\ll||\delta^4||\ll||\delta^6||$, then we say we have a vanishing gradient problem.

If we use Stochastic Gradient Descent, then the size of the change to every parameter $\alpha$ (e.g. a weight, a bias, or any other parameter in more sophisticated networks) in each step taken by the algorithm (we might call this size "the speed of learning of $\alpha$") is proportional to an approximation of $-\frac{\partial C}{\partial\alpha}$ (based on a mini-batch of training examples).

Thus, in case of a vanishing gradient problem, we can say that the speed of learning of parameters of neurons becomes lower and lower, as you move to earlier layers.

So it doesn't necessarily mean that gradients in earlier layers are actually zero, or that they are stuck in any manner, but their speed of learning is low enough to significantly increase the training time, which is why it is called "vanishing gradient problem".

See this answer for a more rigorous explanation of the problem.

Answer 3

Continuing from comments, when you use sigmoid activation function which squashes the input to a small range $(0,1)$, you further multiply it by a small learning rate and more partial derivatives (chain rule) as you go back in layers. The value of delta to be updated diminishes and thus earlier layers get little or no updates. If little, then it would require lot of training. If no, then only changing the activation function (AF) would be of any help. RELUs are currently the best AFs that avoid this problem.

Answer 4

Vanishing Gradient Problem:

• is during backpropagation, a gradient gets smaller and smaller or gets zero, multiplying small gradients together many times as going from output layer to input layer, then a model cannot be trained effectively.
• more easily occurs with more layers in a model.
• is easily caused by Sigmoid activation function which is Sigmoid() because it produces the small values whose ranges are 0<=x<=1, then they are multiplied many times, making a gradient smaller and smaller as going from output layer to input layer.
• can be detected if:
  • parameters significantly change at the layers near output layer whereas parameters slightly change or stay unchanged at the layers near input layer.
  • The weights of the layers near input layer are close to 0 or become 0.
  • convergence is slow or stopped.
• can be mitigated using:
  • Batch Normalization layer.
  • Leaky ReLU activation function. *You can also use ReLU activation function but it sometimes cause Dying ReLU Problem.
  • PReLU activation function.
  • ELU activation function.
  • Gradient Clipping. *Gradient Clipping is the method to keep a gradient in a specified range.
• occurs in:
  • CNN(Convolutional Neural Network).
  • RNN(Recurrent Neural Network).
• doesn't easily occur in:
  • LSTM(Long Short-Term Memory).
  • GRU(Gated Recurrent Unit).
  • Resnet(Residual Neural Network).
  • Transformer.
  • etc.