2
$\begingroup$

In the training of very deep neural nets, the vanishing gradient is the problem that the gradients of weights associated with early layers are too small leading to small overall steps in gradient descent. This seems like a valid problem if the gradient is too small to be represented using available bits. But if this is not the case, then why can't we overcome this problem by having huge learning rates (step sizes) for parameters with small gradient?

Note: This question is different from questions that have already been asked on this on stackexchange as other questions ask about role of activation function in vanishing gradient problem.

$\endgroup$

2 Answers 2

3
$\begingroup$

First, what if the gradient is small because you are close to the true solution? By using a large stepsize, you are going to overshoot.

Second, if the gradient is small then it has a very little directional information due to cancellation, so it would be like going in a random direction with a large stepsize.

$\endgroup$
1
  • $\begingroup$ Can you explain the second point? What is the cancellation you are referring to? $\endgroup$
    – ado sar
    Commented Jun 22, 2023 at 15:57
2
$\begingroup$

The problem is that the gradient is not small for all layers, but small for the deeper layers (further away from the output). For the layers near the output, the gradient may be big.

If you now have a huge learning rate, the weights in the near-to-output layers will shoot around erratically and the net will not converge.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.