In binarized neural networks, where activations are confined to -1/+1 values using the sign function, derivative of the sign function is estimated with a straight-through estimator. However, when I look at the source code for such neural networks, I see that the gradients are clipped for inputs greater than +1 and inputs smaller than -1.

What is the rationale behind doing such clipping and how are those values (-1/+1) chosen? Does it have anything to do with the fact that activations take only -1/+1 values?

Is such clipping useful for different methods of quantization (for activations and/or weights)?


Gradient clipping is a simple trick that prevents gradients form exploding (becoming very large). You don't want to clip too early or too late, this dictates choosing the value to clip. You choose this value based on what are the likely values of gradients and this depends on many factors. There's no deeper rationale behind it other then that it works to solve some particular kind of problems that can arise in some scenarios. It is not related to activations.

So you use it when you have exploding gradient problems and choose such clipping value that prevents the problem.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.