The activation functions I have seen in practice are either sigmoid or tanh. Why isn't step function used? What is bad about using a step function in an activation function for neural networks? What are the effects of using step function? In what ways are sigmoid/tanh superior over step?

  • 12
    $\begingroup$ Assuming you're talking about the heaviside step function, it has 0 gradient everywhere except 0, so you could never do backpropagation since your gradient would always be 0. $\endgroup$ – Alex R. Apr 4 '17 at 1:23
  • $\begingroup$ @AlexR. I think it's a pretty good and simple answer and you should right it as an answer instead of the comment. $\endgroup$ – itdxer Apr 4 '17 at 8:16

There are two main reasons why we cannot use the Heaviside step function in (deep) Neural Net:

  1. At the moment, one of the most efficient ways to train a multi-layer neural network is by using gradient descent with backpropagation. A requirement for backpropagation algorithm is a differentiable activation function. However, the Heaviside step function is non-differentiable at x = 0 and it has 0 derivative elsewhere. This means that gradient descent won’t be able to make a progress in updating the weights.
  2. Recall that the main objective of the neural network is to learn the values of the weights and biases so that the model could produce a prediction as close as possible to the real value. In order to do this, as in many optimisation problems, we’d like a small change in the weight or bias to cause only a small corresponding change in the output from the network. By doing this, we can continuously tweaked the values of weights and bias towards resulting the best approximation. Having a function that can only generate either 0 or 1 (or yes and no), won't help us to achieve this objective.

As answered by the others, the primary reason is that it would not work well during backpropagation. However, adding to what the others wrote, it is important to note that differentiability everywhere is not a necessary condition for backpropagation in neural networks, as one may use subderivatives as well. For example, see the ReLU activation function, which is also non-differentiable at 0 (https://en.wikipedia.org/wiki/Rectifier_(neural_networks))


Why isn't step function used? What is bad about using a step function in an activation function for neural networks?

I assume you mean the Heaviside step function

$$ H(x)= \begin{cases} 1 & x \ge 0 \\ 0 & x< 0 \end{cases}. $$ The key feature of $H$ is not that the gradients are sometimes zero, it's that the gradients are almost always zero.

$H$ has gradient 0 everywhere except at $x=0$. This means that optimizing a model (e.g. neural network) using gradient-based methods because the gradient is almost always zero. (Indeed, its derivative is the Dirac delta function.) This means that the weights will almost always never move because the gradient step has zero length.

Contrasting $H$ to other functions such as ReLU should make it clear why $H$ is unsuitable but other functions with 0 gradient portions can succeed nonetheless.

  • Models using the ReLU have shown marked success, even though the gradient is zero whenever $x<0$. This is because usually not all inputs attain 0 gradient, so weights and biases for $x > 0$ will still update as usual. (However, having 0 gradient "on the left" does give rise to a problem similar to the problems with the Heaviside step function: some weight configurations will always be zero, so these weights are "stuck" and never updated. This is called the dying ReLU phenomenon.)
  • A function having a negligible set where the gradient is not defined is not fatal. The ReLU derivative is not defined at $x=0$ (though the ReLU function is subdifferentiable), but this is inconsequential, both because (1) it rarely happens that floating point arithmetic gives $x=0$ exactly and (2) we can just fudge it by using some number in $[0,1]$ as the gradient for that solitary point -- this arbitrary choice does not make an enormous difference to the final model. (Of course, using a smoother function instead of ReLU will avoid this entirely.)

What are the effects of using step function?

  • There are steep shifts from 0 to 1, which may not fit the data well.
  • The network is not differentiable, so gradient-based training is impossible.

In what ways are sigmoid/tanh superior over step?

The sigmoid layer which combines the affine transformation and the nonlinear activation can be written as $$ \sigma(x) = \frac{1}{1 + \exp(-ax-b)}. $$ For certain $a$, we can view $\sigma$ as a smooth approximation to the step function. A special case of $\sigma$ which has $a$ very large will behave very similarly to $H$, in the sense that there is a rapid increase from 0 to 1, just as we have with $H$.

Likewise, if we need a decreasing function or a constant function, these are also special cases of $\sigma$ for different values of $a$. Successful model training will find $a,b$ that achieve a low loss, i.e. choose the parameters which are shallow or steep as required to fit the data well.

But when using $\sigma$, we still get differentiability, which is the key to training the network.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.