Are all neural network activation functions differentiable? When we design a neural network, we use gradient descent to learn the parameters. Does this require the activation function to be differentiable? 
 A: No! For example, ReLU, which is a widely used activation function, is not differentiable in $z=0$. But they are usually non-differentiable at only a small number of points and they have right derivative and left derivatives at these points. We usually use one of the one-side derivatives. This is rational since digital computers are subject to numerical errors ($z=0$ has been probably some small value rounded to zero). Read chapter 6 of the following book for more details on activation functions:
Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning, MIT Press, 2016, http://deeplearningbook.org
A: If you're going to use gradient descent to learn parameters, you need not only the activation functions to be differential almost everywhere, but ideally the gradient should be non-zero for large parts of the domain. It is not a strict requirement that the gradient be non-0 almost everywhere. For example ReLU has gradient zero for $x \le 0$, and it works pretty well. But whilst the input is in an area of zero gradient, no learning will take place. This manifests in practice in a few ways:


*

*ReLU neurons can get effectively permanently removed from the network, if none of the inputs in the training set ever result in a non-zero gradient. Dropout can sometimes help with this, but not always

*sigmoid activation has gradient close to zero for high and low input values. This is a key incentive for normalizing the mean and variance of data, prior to feeding to the network

*it's one of the driving factors for the use of leaky-ReLU activation, and ELU activation. Both of these have non-zero gradient almost everywhere


("almost everywhere" means with the exception of a finite, or countably infinite, set of points. eg, as Hossein points out, ReLU is not differentiable at $x = 0$)
