# Why are rectified linear units considered non-linear?

Why are activation functions of rectified linear units (ReLU) considered non-linear?

$$f(x) = \max(0,x)$$

They are linear when the input is positive and from my understanding to unlock the representative power of deep networks non-linear activations are a must, otherwise the whole network could be represented by a single layer.

RELUs are nonlinearities. To help your intuition, consider a very simple network with 1 input unit $x$, 2 hidden units $y_i$, and 1 output unit $z$. With this simple network we could implement an absolute value function,

$$z = \max(0, x) + \max(0, -x),$$

or something that looks similar to the commonly used sigmoid function,

$$z = \max(0, x + 1) - \max(0, x - 1).$$

By combining these into larger networks/using more hidden units, we can approximate arbitrary functions.

$\hskip2in$

• Would these types of hand-constructed ReLus be built apriori and hard coded in as layers? If so, how would you know that your network required one of these specially built ReLus in particular? Commented Sep 16, 2016 at 7:53
• @MonicaHeddneck You could specify your own non-linearities, yes. What makes one activation function better than another is a constant research topic. For example, we used to use sigmoids, $\sigma(x) = \frac{1}{1 + e^{-x}}$, but then due to the vanishing gradient problem, ReLUs became more popular. So it's up to you to use different non-linearity activation functions. Commented Sep 19, 2016 at 21:02
• How would you approximate $e^x$ with ReLU in out of sample? Commented Sep 12, 2018 at 21:42
• @Lucas, So basically if combine(+) >1 ReLUs we can approximate any function, but if we simply reLu(reLu(....)) it will be linear always? Also, here you change x to x+1, that could be thought as Z=Wx+b where W & b changes to give different variants of such kind x & x+1?
– Anu
Commented Mar 31, 2019 at 0:12