1
$\begingroup$

In the training of very deep neural nets, the vanishing gradient is the problem that the gradients of weights associated with early layers are too small leading to small overall steps in gradient descent. This seems like a valid problem if the gradient is too small to be represented using available bits. But if this is not the case, then why can't we overcome this problem by having huge learning rates (step sizes) for parameters with small gradient?

Note: This question is different from questions that have already been asked on this on stackexchange as other questions ask about role of activation function in vanishing gradient problem.

$\endgroup$
1
$\begingroup$

The gradient of the loss function changes as a function of the weights, so while the gradient may be a descent direction in the neighborhood of the current iteration, taking a large step may mean that you have moved to no better a solution, i.e. gradient descent diverges.

This is the reason for using small step sizes generally.

$\endgroup$
  • $\begingroup$ If the step function is smooth enough (gradient doesn't change too fast), which may be the case for sigmoid or tanh activation functions in the regime where they have close to zero gradient, the risk of diverging seems low. $\endgroup$ – user2684957 Aug 24 '17 at 6:21
  • $\begingroup$ You're thinking of the gradient of the activation function wrt the inputs. Weight updates are applied according to the gradient of the loss function wrt the weights. $\endgroup$ – Sycorax Aug 24 '17 at 13:18
1
$\begingroup$

First, what if the gradient is small because you are close to the true solution? By using a large stepwise, you are going to overshoot.

Second, if the gradient is small then it has a very little directional information due to cancellation, so it would be like going in a random direction with a large stepsize.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.