When would one want to perform gradient clipping when training a RNN or CNN? I'm especially interested in the latter. What would be a good starting value for clipping? (it can of course be tuned)
$\begingroup$
$\endgroup$
1
-
$\begingroup$ Choosing a good value of gradient clipping depends on the network and the data, so there's no way to know ahead of time what a good choice is. A choice that's too small will make training take a long time, and a choice that is too large won't effectively curb exploding gradient when it happens. stats.stackexchange.com/questions/530288/… $\endgroup$– Sycorax ♦Commented May 19, 2022 at 13:46
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
You would want to perform gradient clipping when you are getting the problem of vanishing gradients or exploding gradients. However, for both scenarios, there are better solutions:
Exploding gradient happens when the gradient becomes too big and you get numerical overflow. This can be easily fixed by initializing the network's weights to smaller values. If this does not work it is likely that there is a bug in the code.
Vanishing gradient happens when the optimization gets stuck in a saddle point, the gradient becomes too small for the optimization to progress. This can be fixed by using gradient descent with momentum or RMS prop or both (also known as the Adam optimizer).
Starting values for the upper bound of gradient clipping would be something smaller than the largest number the variable can take. For the lower bound, I would say it is problem specific but perhaps start with something like 1e-10.
-
2$\begingroup$ I'm not sure if the context of this answer is supposed to exclude RNNs, but if it doesn't, then both of the proposed solutions are not better than gradient clipping especially in the case of RNNs. $\endgroup$– Alex R.Commented Oct 30, 2017 at 19:10
-
$\begingroup$ Sorry I was thinking more in the context of CNNs, feel free to edit $\endgroup$– MiguelCommented Oct 31, 2017 at 9:22