I have seen the word "vanishing gradient" many times in deep learning literature. what is that? gradient respect to what variable? input variable or hidden units?

Does that mean the gradient vector is all zero? Or the optimization stuck in local minima / saddle point?

• You are minimizing error. It is with respect to parameters/coefficients/thetas/weights. – Ketan Sep 4 '17 at 7:15
• @ketan all parameters or a subset of parameters in the middle layer? – Haitao Du Sep 4 '17 at 7:17
• for a neural network, for example, ideally backpropagation should update all the weights right from final layer's weight matrix to the first weight matrix, the one that multiplies with input vector to give the first hidden unit. – Ketan Sep 4 '17 at 7:20
• These notes from Hinton's course explain this problem: cs.toronto.edu/~hinton/csc2535/notes/lec10new.pdf – Jakub Bartczuk Sep 4 '17 at 7:38
• thanks for the link! @JakubBartczuk This is really helpful for me since it also talked both HMM and RNN! See my question here – Haitao Du Sep 4 '17 at 18:00

If you do not carefully choose the range of the initial values for the weights, and if you do not control the range of the values of the weights during training, vanishing gradient would occur which is the main barrier to learning deep networks. The neural networks are trained using the gradient descent algorithm: $$w^{new} := w^{old} - \eta \frac{\partial L}{\partial w}$$ where $L$ is the loss of the network on the current training batch. It is clear that if the $\frac{\partial L}{\partial w}$ is very small, the learning will be very slow, since the changes in $w$ will be very small. So, if the gradients are vanished, the learning will be very very slow.

The reason for vanishing gradient is that during backpropagation, the gradient of early layers (layers near to the input layer) are obtained by multiplying the gradients of later layers (layers near to the output layer). So, for example if the gradients of later layers are less than one, their multiplication vanishes very fast.

• Gradient is the gradient of the loss with respect to each trainable parameters (weights and biases).
• Vanishing gradient does not mean the gradient vector is all zero (except for numerical underflow), but it means the gradients are so small that the learning will be very slow.

Continuing from comments, when you use sigmoid activation function which squashes the input to a small range $(0,1)$, you further multiply it by a small learning rate and more partial derivatives (chain rule) as you go back in layers. The value of delta to be updated diminishes and thus earlier layers get little or no updates. If little, then it would require lot of training. If no, then only changing the activation function (AF) would be of any help. RELUs are currently the best AFs that avoid this problem.

Consider the following feedforward neural network:

• Let $$w^l_{j,k}$$ be the weight for the connection from the $$k^{\text{th}}$$ neuron in the $$(l-1)^{\text{th}}$$ layer to the $$j^{\text{th}}$$ neuron in the $$l^{\text{th}}$$ layer.
• Let $$b^l_j$$ be the bias of the $$j^{\text{th}}$$ neuron in the $$l^{\text{th}}$$ layer.
• Let $$C$$ be the cost function. We consider the inputs and desired outputs of training examples as constants while we train our network, so in our simple network, $$C$$ is a function of the weights and biases in the network. (I.e. weights and biases of hidden layers and the output layer.)

• Let $$\delta^l\equiv\left(\begin{gathered}\frac{\partial C}{\partial w_{1,1}^{l}}\\ \\ \frac{\partial C}{\partial w_{1,2}^{l}}\\ \\ \frac{\partial C}{\partial w_{2,1}^{l}}\\ \\ \frac{\partial C}{\partial w_{2,2}^{l}}\\ \\ \frac{\partial C}{\partial b_{1}^{l}}\\ \\ \frac{\partial C}{\partial b_{2}^{l}} \end{gathered} \right)$$ be "the gradient in the $$l^{\text{th}}$$ layer".

(I use the notation used by Michael Nielsen in the excellent chapter How the backpropagation algorithm works in the book Neural Networks and Deep Learning, except for "the gradient in the $$l^{\text{th}}$$ layer", which I define slightly differently.)

I am not aware of a strict definition of the vanishing gradient problem, but I think Nielsen's definition (from the chapter Why are deep neural networks hard to train? in the same book) is quite clear:

[...] in at least some deep neural networks, the gradient tends to get smaller as we move backward through the hidden layers. This means that neurons in the earlier layers learn much more slowly than neurons in later layers. [...] The phenomenon is known as the vanishing gradient problem.

E.g. in our network, if $$||\delta^2||\ll||\delta^4||\ll||\delta^6||$$, then we say we have a vanishing gradient problem.

If we use Stochastic Gradient Descent, then the size of the change to every parameter $$\alpha$$ (e.g. a weight, a bias, or any other parameter in more sophisticated networks) in each step taken by the algorithm (we might call this size "the speed of learning of $$\alpha$$") is proportional to an approximation of $$-\frac{\partial C}{\partial\alpha}$$ (based on a mini-batch of training examples).

Thus, in case of a vanishing gradient problem, we can say that the speed of learning of parameters of neurons becomes lower and lower, as you move to earlier layers.

So it doesn't necessarily mean that gradients in earlier layers are actually zero, or that they are stuck in any manner, but their speed of learning is low enough to significantly increase the training time, which is why it is called "vanishing gradient problem".

See this answer for a more rigorous explanation of the problem.