4
$\begingroup$

ReLUs are not differentiable at the origin. However, they are widely used in Deep Learning together with Stochastic Gradient Descent algorithms and Backpropagation, where the gradients of the loss function are calculated with the chain rule.

How do these algorithms calculate derivatives given that ReLUs are not differentiable at x=0 ?

$\endgroup$
7
$\begingroup$

At x = 0, the ReLU function is no longer differentiable, however it is sub-differentiable and any value in the range [0,1] is a valid choice of sub-gradient. You may see some implementations simply use 0 sub-gradient at the x = 0 singularity. For further details see the Wikipedia article: Subdervative.

$\endgroup$
  • $\begingroup$ May you kindly expand on what the sub-gradient is? (Apart from reading the wiki, it might be good to give us some insight as to why it works...). Thanks! $\endgroup$ – Creatron Aug 12 '16 at 6:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.