Are all neural network activation functions differentiable?

Question

When we design a neural network, we use gradient descent to learn the parameters. Does this require the activation function to be differentiable?

Answer 1

No! For example, ReLU, which is a widely used activation function, is not differentiable in $z=0$. But they are usually non-differentiable at only a small number of points and they have right derivative and left derivatives at these points. We usually use one of the one-side derivatives. This is rational since digital computers are subject to numerical errors ($z=0$ has been probably some small value rounded to zero). Read chapter 6 of the following book for more details on activation functions:

Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning, MIT Press, 2016, http://deeplearningbook.org

Answer 2

If you're going to use gradient descent to learn parameters, you need not only the activation functions to be differential almost everywhere, but ideally the gradient should be non-zero for large parts of the domain. It is not a strict requirement that the gradient be non-0 almost everywhere. For example ReLU has gradient zero for $x \le 0$, and it works pretty well. But whilst the input is in an area of zero gradient, no learning will take place. This manifests in practice in a few ways:

• ReLU neurons can get effectively permanently removed from the network, if none of the inputs in the training set ever result in a non-zero gradient. Dropout can sometimes help with this, but not always
• sigmoid activation has gradient close to zero for high and low input values. This is a key incentive for normalizing the mean and variance of data, prior to feeding to the network
• it's one of the driving factors for the use of leaky-ReLU activation, and ELU activation. Both of these have non-zero gradient almost everywhere

("almost everywhere" means with the exception of a finite, or countably infinite, set of points. eg, as Hossein points out, ReLU is not differentiable at $x = 0$)