I usually see sigmoid or tanh being used as activation in neural networks. Why not linear slope given that linearity is simpler mathematically?
$\begingroup$
$\endgroup$
1
-
6$\begingroup$ Linear functions are closed under composition, so the result would just be another linear model. Using nonlinear activation functions admits nonlinear models. $\endgroup$– Sycorax ♦Apr 4, 2017 at 2:05
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
As @Sycorax mentioned, linear activation functions cannot solve non-linear problems. The TANH and Sigmoid function introduce this needed non-linearity.
Neural networks have to implement complex mapping functions hence they need activation functions that are non-linear in order to bring in the much needed non-linearity property that enables them to approximate any function. A neuron without an activation function is equivalent to a neuron with a linear activation function given by.
Φ(x)=xΦ(x)=x
Such an activation function adds no non-linearity hence the whole network would just be equivalent to a single linear neuron. That is to say, having a multi-layer linear network is equivalent to one linear node.
Thus it makes no sense building a multi-layer network with linear activation functions, it is better to just have a single node do the job. To make matters worse a single linear node is not capable of dealing with non separable data, that means no matter how large a multi-layer linear network can be it can never solve the classic XOR problem or any other non-linear problem.
Activation functions are also important for squashing the unbounded linearly weighted sum from neurons. This is important to avoid large values accumulating high up the processing hierarchy.
Then lastly, activation functions are decision functions, the ideal decision function is the heaviside step function. But this is not differentiable hence more smoother versions such as the sigmoid function have been used merely because of the fact that they are differentiable which makes them ideal for gradient based optimization algorithms.
$\begingroup$
$\endgroup$
There are piecewise activation functions used where each piece is a linear function and together they form a nonlinear function, such as leaky ReLUs, which turn out to show better performance than sigmoid or tanh in many types of networks.
$\begingroup$
$\endgroup$
Neural network doesn't use linear activation by definition. It's not if it's not possible, linear activation would just give you a new linear transformation from your input variables. So ... why not just fit a linear model?
