What is the derivative of the ReLU activation function defined as:

$$ \mathrm{ReLU}(x) = \mathrm{max}(0, x)$$

What about the special case where there is a discontinuity in the function at $x=0$?


The derivative is:

$$ f(x)= \begin{cases} 0 & \text{if } x < 0 \\ 1 & \text{if } x > 0 \\ \end{cases} $$

And undefined in $x=0$.

The reason for it being undefined at $x=0$ is that its left- and right derivative are not equal.

  • 1
    $\begingroup$ So in practice (implementation), one just picks either $0$ or $1$ for the $x=0$ case? $\endgroup$ – Tom Hale Mar 14 '18 at 9:51
  • 2
    $\begingroup$ The convention is that drdx=1(x>0) $\endgroup$ – neuroguy123 Mar 14 '18 at 13:10
  • $\begingroup$ @TomHale why not use $f(0) = \frac{1}{2}$? The practical implementation is another matter and worth a new question. This is just the mathematics. $\endgroup$ – Jim Mar 14 '18 at 16:17
  • 3
    $\begingroup$ @TomHale by the way, see Nouroz Rahman's answer at quora.com/…: "[...] In my view, in built-in library functions (for example: tf.nn.relu()) derivative at x = 0 is taken zero to ensure a sparser matrix..." $\endgroup$ – Jim Mar 29 '18 at 16:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.