When we design a neural network, we use gradient descent to learn the parameters. Does this require the activation function to be differentiable?


No! For example, ReLU, which is a widely used activation function, is not differentiable in $z=0$. But they are usually non-differentiable at only a small number of points and they have right derivative and left derivatives at these points. We usually use one of the one-side derivatives. This is rational since digital computers are subject to numerical errors ($z=0$ has been probably some small value rounded to zero). Read chapter 6 of the following book for more details on activation functions:

Ian Goodfellow, Yoshua Bengio, and Aaron Courville, Deep Learning, MIT Press, 2016, http://deeplearningbook.org


If you're going to use gradient descent to learn parameters, you need not only the activation functions to be differential almost everywhere, but ideally the gradient should be non-zero for large parts of the domain. It is not a strict requirement that the gradient be non-0 almost everywhere. For example ReLU has gradient zero for $x \le 0$, and it works pretty well. But whilst the input is in an area of zero gradient, no learning will take place. This manifests in practice in a few ways:

  • ReLU neurons can get effectively permanently removed from the network, if none of the inputs in the training set ever result in a non-zero gradient. Dropout can sometimes help with this, but not always
  • sigmoid activation has gradient close to zero for high and low input values. This is a key incentive for normalizing the mean and variance of data, prior to feeding to the network
  • it's one of the driving factors for the use of leaky-ReLU activation, and ELU activation. Both of these have non-zero gradient almost everywhere

("almost everywhere" means with the exception of a finite, or countably infinite, set of points. eg, as Hossein points out, ReLU is not differentiable at $x = 0$)


Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.