# What is the purpose of a neural network activation function?

What is the purpose of a neural network having a non-linear activation function?

Is it correct to say that the non-linear activation function's main purpose is to allow the neural network's decision boundary to be non-linear?

I've read in other StackOverflow answers that the activation function "introduces non-linearity", but that is rather vague.

Another posting states that an answer in the context of deep learning features, but again that is not exactly what I'm asking.

• If your neural network has a linear activation function, then the output, regardless of how many layers or connections, is also going to be linear. So you might as well do a linear regression instead. – Alex R. Aug 4 '16 at 18:31

Is it correct to say that the non-linear activation function's main purpose is to allow the neural network's decision boundary to be non-linear?

Yes.

Neural networks compose several functions in layers: the output of a previous layer is the input to the next layer. If you compose linear functions, these functions are all linear. So the result of stacking several linear functions together is a linear function. Showing this is simple algebra:

\begin{align} \hat{y} &= W_2(W_1x + b_1)+b_2 \\ &= \underbrace{W_2W_1}_W x+\underbrace{W_2b_1+b_2}_b \\ &= Wx+b \end{align}

On the other hand, using a nonlinear function makes the map from the input to the output nonlinear.$$\hat{y} = f(W_2 f(W_1x + b_1)+b_2) \\$$ For some activation function $$f$$, such as $$\tanh$$ or ReLU, this cannot be rewritten as a single linear operation on $$x$$.

Any model which minimizes a loss $$L(y,\hat{y})$$ over parameters $$W_1,W_2,b_1,b_2$$ is equivalent to a model which minimizes the same loss over parameters $$W,b$$. In the case that the loss is the square error loss, this is exactly the same as an OLS model.

If you are estimating this model and observe a discrepancy between an OLS solution and a neural network optimized using gradient descent, it's probably due to either or both of two facts (1) gradient descent is not an effective optimizer for certain problems; (2) the problem is ill-conditioned. For more information, see

This isn't unique to classification problems, either. If you have some sort of regression problem (such as an output that can take on any real number), then using nonlinear activation functions is necessary to model a nonlinear relationship between the inputs and outputs.

For example, a ReLU function's output is either 0 or positive. If the unit is 0, it is effectively "off," so the inputs to the unit are not propagated forward from that function. If the unit is on, the input data is reflected in subsequent layers through that unit. ReLU itself is not linear, and neither is the composition of several layers of several ReLU functions. So the mapping from inputs to classification outcomes is not linear either.

Without activation function many layers would be equivalent to a single layer, as each layer (without an activation function) can be represented by a matrix and a product of many matrices is still a matrix:

$$M = M_1 M_2 \cdots M_n$$

A non-linear activation function and a 2-layer Neural Network can approximate any function. That is why we need to introduce non-linearity, cause we can better approximate.

Role of activation function in neural network:

Before moving towards activation function one must have the basic understanding of neurons in the neural network.

So what does an artificial neuron do? Simply put, it calculates a weighted sum of its input, adds a bias and then decides whether it should be activated or not.

So consider a neuron.

$$Y = \sum (\textit{weight} \cdot \textit{input}) + \textit{bias}$$

Now, the value of $Y$ can be anything ranging from $-\infty$ to $+\infty$. The neuron really doesn’t know the bounds of the value. So how do we decide whether the neuron should activated or not

We decided to add activation functions for this purpose. To check the $Y$ value produced by a neuron and decide whether outside connections should consider this neuron as  activated or not.