The activation functions I have seen in practice are either sigmoid or tanh. Why isn't step function used? What is bad about using a step function in an activation function for neural networks? What are the effects of using step function? In what ways are sigmoid/tanh superior over step?

  • 5
    $\begingroup$ Assuming you're talking about the heaviside step function, it has 0 gradient everywhere except 0, so you could never do backpropagation since your gradient would always be 0. $\endgroup$ – Alex R. Apr 4 '17 at 1:23
  • $\begingroup$ @AlexR. I think it's a pretty good and simple answer and you should right it as an answer instead of the comment. $\endgroup$ – itdxer Apr 4 '17 at 8:16

There are two main reasons why we cannot use the Heaviside step function in (deep) Neural Net:

  1. At the moment, one of the most efficient ways to train a multi-layer neural network is by using gradient descent with backpropagation. A requirement for backpropagation algorithm is a differentiable activation function. However, the Heaviside step function is non-differentiable at x = 0 and it has 0 derivative elsewhere. This means that gradient descent won’t be able to make a progress in updating the weights.
  2. Recall that the main objective of the neural network is to learn the values of the weights and biases so that the model could produce a prediction as close as possible to the real value. In order to do this, as in many optimisation problems, we’d like a small change in the weight or bias to cause only a small corresponding change in the output from the network. By doing this, we can continuously tweaked the values of weights and bias towards resulting the best approximation. Having a function that can only generate either 0 or 1 (or yes and no), won't help us to achieve this objective.

As answered by the others, the primary reason is that it would not work well during backpropagation. However, adding to what the others wrote, it is important to note that differentiability everywhere is not a necessary condition for backpropagation in neural networks, as one may use subderivatives as well. For example, see the ReLU activation function, which is also non-differentiable at 0 (https://en.wikipedia.org/wiki/Rectifier_(neural_networks))


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.