The activation functions I have seen in practice are either sigmoid or tanh. Why isn't step function used? What is bad about using a step function in an activation function for neural networks? What are the effects of using step function? In what ways are sigmoid/tanh superior over step?
There are two main reasons why we cannot use the Heaviside step function in (deep) Neural Net:
- At the moment, one of the most efficient ways to train a multi-layer neural network is by using gradient descent with backpropagation. A requirement for backpropagation algorithm is a differentiable activation function. However, the Heaviside step function is non-differentiable at x = 0 and it has 0 derivative elsewhere. This means that gradient descent won’t be able to make a progress in updating the weights.
- Recall that the main objective of the neural network is to learn the values of the weights and biases so that the model could produce a prediction as close as possible to the real value. In order to do this, as in many optimisation problems, we’d like a small change in the weight or bias to cause only a small corresponding change in the output from the network. By doing this, we can continuously tweaked the values of weights and bias towards resulting the best approximation. Having a function that can only generate either 0 or 1 (or yes and no), won't help us to achieve this objective.
As answered by the others, the primary reason is that it would not work well during backpropagation. However, adding to what the others wrote, it is important to note that differentiability everywhere is not a necessary condition for backpropagation in neural networks, as one may use subderivatives as well. For example, see the ReLU activation function, which is also non-differentiable at 0 (https://en.wikipedia.org/wiki/Rectifier_(neural_networks))