34
$\begingroup$

Why are activation functions of rectified linear units (ReLU) considered non-linear?

$$ f(x) = \max(0,x)$$

They are linear when the input is positive and from my understanding to unlock the representative power of deep networks non-linear activations are a must, otherwise the whole network could be represented by a single layer.

Improve this question
$\endgroup$
1

1 Answer 1

Reset to default
41
$\begingroup$

RELUs are nonlinearities. To help your intuition, consider a very simple network with 1 input unit $x$, 2 hidden units $y_i$, and 1 output unit $z$. With this simple network we could implement an absolute value function,

$$z = \max(0, x) + \max(0, -x),$$

or something that looks similar to the commonly used sigmoid function,

$$z = \max(0, x + 1) - \max(0, x - 1).$$

By combining these into larger networks/using more hidden units, we can approximate arbitrary functions.

$\hskip2in$RELU network function

Improve this answer
$\endgroup$
4
  • $\begingroup$ Would these types of hand-constructed ReLus be built apriori and hard coded in as layers? If so, how would you know that your network required one of these specially built ReLus in particular? $\endgroup$
    – Monica Heddneck
    Commented Sep 16, 2016 at 7:53
  • 6
    $\begingroup$ @MonicaHeddneck You could specify your own non-linearities, yes. What makes one activation function better than another is a constant research topic. For example, we used to use sigmoids, $\sigma(x) = \frac{1}{1 + e^{-x}}$, but then due to the vanishing gradient problem, ReLUs became more popular. So it's up to you to use different non-linearity activation functions. $\endgroup$
    – Tarin Ziyaee
    Commented Sep 19, 2016 at 21:02
  • 2
    $\begingroup$ How would you approximate $e^x$ with ReLU in out of sample? $\endgroup$
    – Aksakal
    Commented Sep 12, 2018 at 21:42
  • 2
    $\begingroup$ @Lucas, So basically if combine(+) >1 ReLUs we can approximate any function, but if we simply reLu(reLu(....)) it will be linear always? Also, here you change x to x+1, that could be thought as Z=Wx+b where W & b changes to give different variants of such kind x & x+1? $\endgroup$
    – Anu
    Commented Mar 31, 2019 at 0:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.