ReLu vs a linear activation function

Question

I read this great answer about how ReLu could approximate non-linear functions. However, if ReLu can approximate such functions, why cant linear activation function do the same? If they can, why linear activation functions are strictly prohibited in NN?

How does the Rectified Linear Unit (ReLU) activation function produce non-linear interaction of its inputs?

Answer 1

Consider a simple multilayer perceptron (feedforward neural network) with one hidden layer that accepts $p$ inputs, has $q$ hidden units, a hidden activation function $\sigma$, and one output with a linear activation: $$ \widehat{f}(\mathbf{x}) = b + \sum_{i=1}^q u_i \sigma(a_i + \mathbf{w}_i \cdot \mathbf{x}) $$ (the parameters to be learned here are the $a_i$'s, the $\mathbf{w}_i$'s, the $u_i$'s, and $b$). Now suppose $\sigma$ is a linear activation function: without loss of generality, suppose $\sigma(x) = x$. Then we have $$ \begin{aligned} \widehat{f}(\mathbf{x}) &= b + \sum_{i=1}^q u_i \sigma(a_i + \mathbf{w}_i \cdot \mathbf{x}) \\ &= b + \sum_{i=1}^q u_i (a_i + \mathbf{w}_i \cdot \mathbf{x}) \\ &= b + \sum_{i=1}^q (u_i a_i + u_i \mathbf{w}_i \cdot \mathbf{x}) \\ &= b + \sum_{i=1}^q u_i a_i + \sum_{i=1}^q (u_i \mathbf{w}_i \cdot \mathbf{x}) \\ &= \left(b + \sum_{i=1}^q u_i a_i\right) + \left(\sum_{i=1}^q u_i \mathbf{w}_i\right) \cdot \mathbf{x} \\ \\ &= b^\prime + \mathbf{w}^\prime \cdot \mathbf{x} \end{aligned} $$ where $$ \begin{aligned} b^\prime &= b + \sum_{i=1}^q u_i a_i, & \mathbf{w}^\prime &= \sum_{i=1}^q u_i \mathbf{w}_i. \end{aligned} $$ Thus, with a linear activation function, we've reduced the multilayer perceptron to a linear model. The takeaway is that there is no benefit to depth in a multilayer perceptron with a linear activation function. This is because the composition of two affine functions is just another affine function.