# What is the derivative of the ReLU activation function?

What is the derivative of the ReLU activation function defined as:

$$\mathrm{ReLU}(x) = \mathrm{max}(0, x)$$

What about the special case where there is a discontinuity in the function at $x=0$?

## 1 Answer

The derivative is:

$$f(x)= \begin{cases} 0 & \text{if } x < 0 \\ 1 & \text{if } x > 0 \\ \end{cases}$$

And undefined in $x=0$.

The reason for it being undefined at $x=0$ is that its left- and right derivative are not equal.

• So in practice (implementation), one just picks either $0$ or $1$ for the $x=0$ case? – Tom Hale Mar 14 '18 at 9:51
• The convention is that drdx=1(x>0) – neuroguy123 Mar 14 '18 at 13:10
• @TomHale why not use $f(0) = \frac{1}{2}$? The practical implementation is another matter and worth a new question. This is just the mathematics. – Jim Mar 14 '18 at 16:17
• @TomHale by the way, see Nouroz Rahman's answer at quora.com/…: "[...] In my view, in built-in library functions (for example: tf.nn.relu()) derivative at x = 0 is taken zero to ensure a sparser matrix..." – Jim Mar 29 '18 at 16:17