When would one want to perform gradient clipping when training a RNN or CNN? I'm especially interested in the latter. What would be a good starting value for clipping? (it can of course be tuned)


You would want to perform gradient clipping when you are getting the problem of vanishing gradients or exploding gradients. However, for both scenarios, there are better solutions:

  • Exploding gradient happens when the gradient becomes too big and you get numerical overflow. This can be easily fixed by initializing the network's weights to smaller values. If this does not work it is likely that there is a bug in the code.

  • Vanishing gradient happens when the optimization gets stuck in a saddle point, the gradient becomes too small for the optimization to progress. This can be fixed by using gradient descent with momentum or RMS prop or both (also known as the Adam optimizer).

Starting values for the upper bound of gradient clipping would be something smaller than the largest number the variable can take. For the lower bound, I would say it is problem specific but perhaps start with something like 1e-10.

  • 2
    $\begingroup$ I'm not sure if the context of this answer is supposed to exclude RNNs, but if it doesn't, then both of the proposed solutions are not better than gradient clipping especially in the case of RNNs. $\endgroup$
    – Alex R.
    Oct 30 '17 at 19:10
  • $\begingroup$ Sorry I was thinking more in the context of CNNs, feel free to edit $\endgroup$
    – Miguel
    Oct 31 '17 at 9:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.