When would one want to perform gradient clipping when training a RNN or CNN? I'm especially interested in the latter. What would be a good starting value for clipping? (it can of course be tuned)
You would want to perform gradient clipping when you are getting the problem of vanishing gradients or exploding gradients. However, for both scenarios, there are better solutions:
Exploding gradient happens when the gradient becomes too big and you get numerical overflow. This can be easily fixed by initializing the network's weights to smaller values. If this does not work it is likely that there is a bug in the code.
Vanishing gradient happens when the optimization gets stuck in a saddle point, the gradient becomes too small for the optimization to progress. This can be fixed by using gradient descent with momentum or RMS prop or both (also known as the Adam optimizer).
Starting values for the upper bound of gradient clipping would be something smaller than the largest number the variable can take. For the lower bound, I would say it is problem specific but perhaps start with something like 1e-10.