# ReLu vs a linear activation function

I read this great answer about how ReLu could approximate non-linear functions. However, if ReLu can approximate such functions, why cant linear activation function do the same? If they can, why linear activation functions are strictly prohibited in NN?

How does the Rectified Linear Unit (ReLU) activation function produce non-linear interaction of its inputs?

• Linear functions are closed under composition.
– Sycorax
Feb 11, 2019 at 17:23
• @Sycorax, Thanks for the response, I see that you are the person who answered in the link above. May I ask for further clarification for that answer? In that answer, you have h2(x)=g(x)+g(−x)+g(2x−2)+g(−2x+2). Let just focus on the positive side of the ReLu, i.e. h2'(x) = g(x) + g(2x-2). The main idea here is that for the range of x from [0-1], your slope is 1. From range [1-2], your slope is 3, hence, nonlinearity. But you do need to impose the range on the two functions right? Otherwise, are they just linear combination, and therefore, it is closed under composition too: h2'(x) = 3x-2? Feb 11, 2019 at 17:59
• Feb 11, 2019 at 18:06
• @Hyperloop I don't understand your comment. Compositions of ReLUs are piecewise linear. Piecewise linear functions are linear only on specific intervals. $h_2$ is nonlinear for real inputs.
– Sycorax
Feb 11, 2019 at 18:22
• Ok, I know why I was confused now. In fact you don't need to specify any intervals because ReLu only active for g(2x-2) when x>1. Therefore, you can have a different slope when x>1. Clear now, thanks! Feb 11, 2019 at 19:34

Consider a simple multilayer perceptron (feedforward neural network) with one hidden layer that accepts $$p$$ inputs, has $$q$$ hidden units, a hidden activation function $$\sigma$$, and one output with a linear activation: $$\widehat{f}(\mathbf{x}) = b + \sum_{i=1}^q u_i \sigma(a_i + \mathbf{w}_i \cdot \mathbf{x})$$ (the parameters to be learned here are the $$a_i$$'s, the $$\mathbf{w}_i$$'s, the $$u_i$$'s, and $$b$$). Now suppose $$\sigma$$ is a linear activation function: without loss of generality, suppose $$\sigma(x) = x$$. Then we have \begin{aligned} \widehat{f}(\mathbf{x}) &= b + \sum_{i=1}^q u_i \sigma(a_i + \mathbf{w}_i \cdot \mathbf{x}) \\ &= b + \sum_{i=1}^q u_i (a_i + \mathbf{w}_i \cdot \mathbf{x}) \\ &= b + \sum_{i=1}^q (u_i a_i + u_i \mathbf{w}_i \cdot \mathbf{x}) \\ &= b + \sum_{i=1}^q u_i a_i + \sum_{i=1}^q (u_i \mathbf{w}_i \cdot \mathbf{x}) \\ &= \left(b + \sum_{i=1}^q u_i a_i\right) + \left(\sum_{i=1}^q u_i \mathbf{w}_i\right) \cdot \mathbf{x} \\ \\ &= b^\prime + \mathbf{w}^\prime \cdot \mathbf{x} \end{aligned} where \begin{aligned} b^\prime &= b + \sum_{i=1}^q u_i a_i, & \mathbf{w}^\prime &= \sum_{i=1}^q u_i \mathbf{w}_i. \end{aligned} Thus, with a linear activation function, we've reduced the multilayer perceptron to a linear model. The takeaway is that there is no benefit to depth in a multilayer perceptron with a linear activation function. This is because the composition of two affine functions is just another affine function.
• @Hyperloop while ReLU is linear on the intervals $(-\infty,0)$ and $(0,\infty)$, it is not linear globally. This nonlinearity is what allows networks with ReLU activations to benefit from added depth: they do not reduce to linear models Feb 11, 2019 at 17:41