Why is step function not used in activation functions in machine learning?

Question

The activation functions I have seen in practice are either sigmoid or tanh. Why isn't step function used? What is bad about using a step function in an activation function for neural networks? What are the effects of using step function? In what ways are sigmoid/tanh superior over step?

Answer 1

There are two main reasons why we cannot use the Heaviside step function in (deep) Neural Net:

1. At the moment, one of the most efficient ways to train a multi-layer neural network is by using gradient descent with backpropagation. A requirement for backpropagation algorithm is a differentiable activation function. However, the Heaviside step function is non-differentiable at x = 0 and it has 0 derivative elsewhere. This means that gradient descent won’t be able to make a progress in updating the weights.
2. Recall that the main objective of the neural network is to learn the values of the weights and biases so that the model could produce a prediction as close as possible to the real value. In order to do this, as in many optimisation problems, we’d like a small change in the weight or bias to cause only a small corresponding change in the output from the network. By doing this, we can continuously tweaked the values of weights and bias towards resulting the best approximation. Having a function that can only generate either 0 or 1 (or yes and no), won't help us to achieve this objective.

Answer 2

As answered by the others, the primary reason is that it would not work well during backpropagation. However, adding to what the others wrote, it is important to note that differentiability everywhere is not a necessary condition for backpropagation in neural networks, as one may use subderivatives as well. For example, see the ReLU activation function, which is also non-differentiable at 0 (https://en.wikipedia.org/wiki/Rectifier_(neural_networks))

Answer 3

Why isn't step function used? What is bad about using a step function in an activation function for neural networks?

I assume you mean the Heaviside step function

$$ H(x)= \begin{cases} 1 & x \ge 0 \\ 0 & x< 0 \end{cases}. $$ The key feature of $H$ is not that the gradients are sometimes zero, it's that the gradients are almost always zero.

$H$ has gradient 0 everywhere except at $x=0$. This means that optimizing a model (e.g. neural network) using gradient-based methods because the gradient is almost always zero. (Indeed, its derivative is the Dirac delta function.) This means that the weights will almost always never move because the gradient step has zero length.

Contrasting $H$ to other functions such as ReLU should make it clear why $H$ is unsuitable but other functions with 0 gradient portions can succeed nonetheless.

• Models using the ReLU have shown marked success, even though the gradient is zero whenever $x<0$. This is because usually not all inputs attain 0 gradient, so weights and biases for $x > 0$ will still update as usual. (However, having 0 gradient "on the left" does give rise to a problem similar to the problems with the Heaviside step function: some weight configurations will always be zero, so these weights are "stuck" and never updated. This is called the dying ReLU phenomenon.)
• A function having a negligible set where the gradient is not defined is not fatal. The ReLU derivative is not defined at $x=0$ (though the ReLU function is subdifferentiable), but this is inconsequential, both because (1) it rarely happens that floating point arithmetic gives $x=0$ exactly and (2) we can just fudge it by using some number in $[0,1]$ as the gradient for that solitary point -- this arbitrary choice does not make an enormous difference to the final model. (Of course, using a smoother function instead of ReLU will avoid this entirely.)

What are the effects of using step function?

• There are steep shifts from 0 to 1, which may not fit the data well.
• The network is not differentiable, so gradient-based training is impossible.

In what ways are sigmoid/tanh superior over step?

The sigmoid layer which combines the affine transformation and the nonlinear activation can be written as $$ \sigma(x) = \frac{1}{1 + \exp(-ax-b)}. $$ For certain $a$, we can view $\sigma$ as a smooth approximation to the step function. A special case of $\sigma$ which has $a$ very large will behave very similarly to $H$, in the sense that there is a rapid increase from 0 to 1, just as we have with $H$.

Likewise, if we need a decreasing function or a constant function, these are also special cases of $\sigma$ for different values of $a$. Successful model training will find $a,b$ that achieve a low loss, i.e. choose the parameters which are shallow or steep as required to fit the data well.

But when using $\sigma$, we still get differentiability, which is the key to training the network.

Answer 4

---

Since we differentiate the activation function in back propagation process to find optimal weight values, we need to have an activation function that is suitable for differentiation.

There mainly 2 types of activation functions:

*Linear Functions

*Non Linear Functions

Linear Functions:

1.Identity function:f(x)=x, f'(x)=1

It is too simple

2.Step function:f(x)=1 if x>=0, f(x)=0 if x<0

It is discontinous so cannot be differentiated

Also linear functions only work on linearly separable inputs.

So we use non linear functions like sigmoid,tanh,ReLU which are continous and have good differentiation results.

Answer 5

The cons(disadvantages) to use step function in neural networks are more than the commonly used activation functions such as ReLU, Sigmoid, Tanh activation functions, etc as shown below:

Step function:

1. which can only express the two values 0 and 1 creates the model of bad accuracy, predicting inaccurately so the activation functions which can express wider values are needed to create the model of good accuracy, predicting accurately.

2. causes Dying ReLU Problem, producing zero for negative input values. *Dying ReLU Problem is during backpropagation, once the nodes(neurons) with ReLU activation function recieve negative input values, they always produce zero for any input values, finally, they are never recovered to produce any values except zero, then a model cannot be trained effectively.

3. is non-differentiable at x=0 which means the gradient for step function doesn't exist at x=0 during backpropagation. *Backpropagation does differential to calculate and get a gradient.