I read this great answer about how ReLu could approximate non-linear functions. However, if ReLu can approximate such functions, why cant linear activation function do the same? If they can, why linear activation functions are strictly prohibited in NN?

How does the Rectified Linear Unit (ReLU) activation function produce non-linear interaction of its inputs?

  • $\begingroup$ Linear functions are closed under composition. $\endgroup$
    – Sycorax
    Feb 11, 2019 at 17:23
  • $\begingroup$ @Sycorax, Thanks for the response, I see that you are the person who answered in the link above. May I ask for further clarification for that answer? In that answer, you have h2(x)=g(x)+g(−x)+g(2x−2)+g(−2x+2). Let just focus on the positive side of the ReLu, i.e. h2'(x) = g(x) + g(2x-2). The main idea here is that for the range of x from [0-1], your slope is 1. From range [1-2], your slope is 3, hence, nonlinearity. But you do need to impose the range on the two functions right? Otherwise, are they just linear combination, and therefore, it is closed under composition too: h2'(x) = 3x-2? $\endgroup$
    – Hyperloop
    Feb 11, 2019 at 17:59
  • 3
    $\begingroup$ Possible duplicate of Why are rectified linear units considered non-linear? stats.stackexchange.com/questions/299915/… $\endgroup$
    – Aksakal
    Feb 11, 2019 at 18:06
  • $\begingroup$ @Hyperloop I don't understand your comment. Compositions of ReLUs are piecewise linear. Piecewise linear functions are linear only on specific intervals. $h_2$ is nonlinear for real inputs. $\endgroup$
    – Sycorax
    Feb 11, 2019 at 18:22
  • $\begingroup$ Ok, I know why I was confused now. In fact you don't need to specify any intervals because ReLu only active for g(2x-2) when x>1. Therefore, you can have a different slope when x>1. Clear now, thanks! $\endgroup$
    – Hyperloop
    Feb 11, 2019 at 19:34

1 Answer 1


Consider a simple multilayer perceptron (feedforward neural network) with one hidden layer that accepts $p$ inputs, has $q$ hidden units, a hidden activation function $\sigma$, and one output with a linear activation: $$ \widehat{f}(\mathbf{x}) = b + \sum_{i=1}^q u_i \sigma(a_i + \mathbf{w}_i \cdot \mathbf{x}) $$ (the parameters to be learned here are the $a_i$'s, the $\mathbf{w}_i$'s, the $u_i$'s, and $b$). Now suppose $\sigma$ is a linear activation function: without loss of generality, suppose $\sigma(x) = x$. Then we have $$ \begin{aligned} \widehat{f}(\mathbf{x}) &= b + \sum_{i=1}^q u_i \sigma(a_i + \mathbf{w}_i \cdot \mathbf{x}) \\ &= b + \sum_{i=1}^q u_i (a_i + \mathbf{w}_i \cdot \mathbf{x}) \\ &= b + \sum_{i=1}^q (u_i a_i + u_i \mathbf{w}_i \cdot \mathbf{x}) \\ &= b + \sum_{i=1}^q u_i a_i + \sum_{i=1}^q (u_i \mathbf{w}_i \cdot \mathbf{x}) \\ &= \left(b + \sum_{i=1}^q u_i a_i\right) + \left(\sum_{i=1}^q u_i \mathbf{w}_i\right) \cdot \mathbf{x} \\ \\ &= b^\prime + \mathbf{w}^\prime \cdot \mathbf{x} \end{aligned} $$ where $$ \begin{aligned} b^\prime &= b + \sum_{i=1}^q u_i a_i, & \mathbf{w}^\prime &= \sum_{i=1}^q u_i \mathbf{w}_i. \end{aligned} $$ Thus, with a linear activation function, we've reduced the multilayer perceptron to a linear model. The takeaway is that there is no benefit to depth in a multilayer perceptron with a linear activation function. This is because the composition of two affine functions is just another affine function.

  • $\begingroup$ Understood that, and thanks for the answer. But my question is really about why ReLu (which is a linear function when z>0) can approximate a non-linear function, and a linear activation function can not? It's not much about why a linear activation function is prohibited for MLP or NN in general. Any linear function could be approximated by linear function, for example, h(x) = g(x) + g(2*x) + g(3*x) + g(-x) ... You just need to add more terms in there, don't you? $\endgroup$
    – Hyperloop
    Feb 11, 2019 at 17:35
  • 1
    $\begingroup$ @Hyperloop while ReLU is linear on the intervals $(-\infty,0)$ and $(0,\infty)$, it is not linear globally. This nonlinearity is what allows networks with ReLU activations to benefit from added depth: they do not reduce to linear models $\endgroup$ Feb 11, 2019 at 17:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.