What is the purpose of a neural network activation function?

Question

What is the purpose of a neural network having a non-linear activation function?

Is it correct to say that the non-linear activation function's main purpose is to allow the neural network's decision boundary to be non-linear?

I've read in other StackOverflow answers that the activation function "introduces non-linearity", but that is rather vague.

Another posting states that an answer in the context of deep learning features, but again that is not exactly what I'm asking.

Answer 1

Is it correct to say that the non-linear activation function's main purpose is to allow the neural network's decision boundary to be non-linear?

Yes.

Neural networks compose several functions in layers: the output of a previous layer is the input to the next layer. Linear functions are closed under composition, so the result of stacking several linear functions together is a linear function:

$$ \begin{align} \hat{y} &= W_2(W_1x + b_1)+b_2 \\ &= \underbrace{W_2W_1}_W x+\underbrace{W_2b_1+b_2}_b \\ &= Wx+b \end{align} $$

Any model which minimizes a loss $L(y,\hat{y})$ over parameters $W_1,W_2,b_1,b_2$ is equivalent to a model which minimizes the same loss over parameters $W,b$. In the special case that the loss is the square error loss, this is exactly the same as an OLS model.

On the other hand, using a nonlinear function $f$ makes the map from the input to the output nonlinear:$$ \hat{y} = f(W_2 f(W_1x + b_1)+b_2) \\ $$ When $f$ is some well-chosen pointwise nonlinear function, such as $\tanh$ or ReLU, this cannot be rewritten as a single linear operation on $x.$

The importance of nonlinearity to neural networks isn't unique to classification problems. If you have some sort of regression problem (such as an output that can take on any real number), then using nonlinear activation functions is necessary to model a nonlinear relationship between the inputs and outputs.

In both classification and regression settings, the purpose of an activation function is akin to basis expansion or regression splines, with the added flexibility that the neural network optimizes the weights to create the basis & reduce the loss.

Answer 2

Without activation function many layers would be equivalent to a single layer, as each layer (without an activation function) can be represented by a matrix and a product of many matrices is still a matrix:

$$ M = M_1 M_2 \cdots M_n$$

Answer 3

Role of activation function in neural network:

Before moving towards activation function one must have the basic understanding of neurons in the neural network.

So what does an artificial neuron do? Simply put, it calculates a weighted sum of its input, adds a bias and then decides whether it should be activated or not.

So consider a neuron.

$$ Y = \sum (\textit{weight} \cdot \textit{input}) + \textit{bias} $$

Now, the value of $Y$ can be anything ranging from $-\infty$ to $+\infty$. The neuron really doesn’t know the bounds of the value. So how do we decide whether the neuron should activated or not

We decided to add activation functions for this purpose. To check the $Y$ value produced by a neuron and decide whether outside connections should consider this neuron as  activated or not.

Answer 4

A non-linear activation function and a 2-layer Neural Network can approximate any function. That is why we need to introduce non-linearity, cause we can better approximate.

Answer 5

Let me try to answer this without giving any equations.

Q) What is the purpose of a neural network having a non-linear activation function?

It is not necessary to add non-linear activation function to a neural net if the situation being modelled is linear. However in reality, complicated relationships cannot be represented by a straight line. Most real-world problems are non-linear. In order to help the model converge upon the solution for such cases, we need to bring in an element of non-linearity. One of the ways this can be done is to add functions/layers that are non-linear. Now the model has more power to find solutions

Q) Is it correct to say that the non-linear activation function's main purpose is to allow the neural network's decision boundary to be non-linear?

Yes that is the only purpose as explained above. The decision boundary now need not be a simple straight line but can be a circle or any other complicated shape. Now that you have provided the model with great deal of flexibility to 'fit' solutions, you should be careful and take precautions against overfitting

Answer 6

An activation function is the function or layer which enables neural network to learn complex(non-linear) relationships by transforming the output of the previous layer. *Without activation functions, neural network can only learn linear relationships.

There are popular activation functions as shown below:

(1) Step Function:

• can convert input values to 0 or 1. *If input < 0, then 0 while if input >= 0, then 1.
• is also called Binary Step Function, Unit Step Function, Binary Threshold Function, Threshold Function, Heaviside Step Function or Heaviside Function.
• is heaviside() in PyTorch.

(2) ReLU(Rectified Linear Unit) Function:

• can convert input values to the output values between 0 and input. *If input <= 0, then 0 while if input > 0, then input.
• is ReLU() in PyTorch.

(3) Leaky ReLU(Leaky Rectified Linear Unit) Function:

• can convert input values to the output values between input * slope and input. *Memos:
  • If input <= 0, then input * slope while if input > 0, then input.
  • slope is 0.01 basically.
• is the improved version of ReLU Function.
• is LeakyReLU() in PyTorch.

(4) ELU(Exponential Linear Unit) Function:

• can convert input values to the output values between a * (e^input - 1) and input. *Memos:
  • If input <= 0, then a * (e^input - 1) while if input > 0, then input.
  • a is 1.0 basically.
• is the improved version of ReLU Function.
• is ELU() in PyTorch.

(5) Sigmoid Function:

• can convert input values to the output values between 0 and 1.
• 's formula is y = 1 / (1 + e^-x).
• is Sigmoid() in PyTorch.

(6) Tanh Function:

• can convert input values to the output values between -1 and 1.
• 's formula is y = (e^x - e^-x) / (e^x + e^-x).
• is also called Hyperbolic Tangent Function.
• is Tanh() in PyTorch.

(7) Softmax Function:

• can convert input values to the output values between 0 and 1 each and whose sum is 1(100%). *If input values are [5, 4, -1], then the output values are [0.730, 0.268, 0.002] which is 0.730(73%) + 0.268(26.8%) + 0.002(0.2%) = 1(100%).
• 's formula is:
• is Softmax() in PyTorch.