Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
What is the purpose of a neural network having a non-linear activation function?
Is it correct to say that the non-linear activation function's main purpose is to allow the neural network's decision boundary to be non-linear?
I've read in other StackOverflow answers that the activation function "introduces non-linearity", but that is rather vague.
Another posting states that an answer in the context of deep learning features, but again that is not exactly what I'm asking.
Is it correct to say that the non-linear activation function's main purpose is to allow the neural network's decision boundary to be non-linear?
Yes.
Neural networks compose several functions in layers: the output of a previous layer is the input to the next layer. If you compose linear functions, these functions are all linear. So the result of stacking several linear functions together is a linear function. Showing this is simple algebra:
$$ \begin{align} \hat{y} &= W_2(W_1x + b_1)+b_2 \\ &= \underbrace{W_2W_1}_W x+\underbrace{W_2b_1+b_2}_b \\ &= Wx+b \end{align} $$
Any model which minimizes a loss $L(y,\hat{y})$ over parameters $W_1,W_2,b_1,b_2$ is equivalent to a model which minimizes the same loss over parameters $W,b$. In the case that the loss is the square error loss, this is exactly the same as an OLS model.
On the other hand, using a nonlinear function makes the map from the input to the output nonlinear.$$ \hat{y} = f(W_2 f(W_1x + b_1)+b_2) \\ $$ For some activation function $f$, such as $\tanh$ or ReLU, this cannot be rewritten as a single linear operation on $x$.
If you are estimating this model and observe a discrepancy between an OLS solution and a neural network optimized using gradient descent, it's probably due to either or both of two facts (1) gradient descent is not an effective optimizer for certain problems; (2) the problem is ill-conditioned. For more information, see
Do we need gradient descent to find the coefficients of a linear regression model?
Why use gradient descent for linear regression, when a closed-form math solution is available?
This isn't unique to classification problems, either. If you have some sort of regression problem (such as an output that can take on any real number), then using nonlinear activation functions is necessary to model a nonlinear relationship between the inputs and outputs.
For example, a ReLU function's output is either 0 or positive. If the unit is 0, it is effectively "off," so the inputs to the unit are not propagated forward from that function. If the unit is on, the input data is reflected in subsequent layers through that unit. ReLU itself is not linear, and neither is the composition of several layers of several ReLU functions. So the mapping from inputs to classification outcomes is not linear either.
Without activation function many layers would be equivalent to a single layer, as each layer (without an activation function) can be represented by a matrix and a product of many matrices is still a matrix:
$$ M = M_1 M_2 \cdots M_n$$
Role of activation function in neural network:
Before moving towards activation function one must have the basic understanding of neurons in the neural network.
So what does an artificial neuron do? Simply put, it calculates a weighted sum of its input, adds a bias and then decides whether it should be activated or not.
So consider a neuron.
$$ Y = \sum (\textit{weight} \cdot \textit{input}) + \textit{bias} $$
Now, the value of $Y$ can be anything ranging from $-\infty$ to $+\infty$. The neuron really doesn’t know the bounds of the value. So how do we decide whether the neuron should activated or not
We decided to add activation functions for this purpose. To check the $Y$ value produced by a neuron and decide whether outside connections should consider this neuron as activated or not.
A non-linear activation function and a 2-layer Neural Network can approximate any function. That is why we need to introduce non-linearity, cause we can better approximate.
Let me try to answer this without giving any equations.
Q) What is the purpose of a neural network having a non-linear activation function?
It is not necessary to add non-linear activation function to a neural net if the situation being modelled is linear. However in reality, complicated relationships cannot be represented by a straight line. Most real-world problems are non-linear. In order to help the model converge upon the solution for such cases, we need to bring in an element of non-linearity. One of the ways this can be done is to add functions/layers that are non-linear. Now the model has more power to find solutions
Q) Is it correct to say that the non-linear activation function's main purpose is to allow the neural network's decision boundary to be non-linear?
Yes that is the only purpose as explained above. The decision boundary now need not be a simple straight line but can be a circle or any other complicated shape. Now that you have provided the model with great deal of flexibility to 'fit' solutions, you should be careful and take precautions against overfitting
