When used as an activation function in deep neural networks The ReLU function outperforms other non-linear functions like tanh or sigmoid . In my understanding the whole purpose of an activation function is to let the weighted inputs to a neuron interact non-linearly. For example, when using $sin(z)$ as the activation, the output of a two input neuron would be:

$$ sin(w_0+w_1*x_1+w_2*x_2) $$

which would approximate the function $$ (w_0+w_1*x_1+w_2*x_2) - {(w_0+w_1*x_1+w_2*x_2)^3 \over 6} + {(w_0+w_1*x_1+w_2*x_2)^5 \over 120} $$

and contain all kinds of combinations of different powers of the features $x_1$ and $x_2$.

Although the ReLU is also technically a non-linear function, I don't see how it can produce non-linear terms like the $sin(), tanh()$ and other activations do.

Edit: Although my question is similar to this question, I'd like to know how even a cascade of ReLUs are able to approximate such non-linear terms.

  • 2
    $\begingroup$ IMHO this question is much more clearly stated than the one it's marked as a duplicate of. So then to ask his question the OP would have to edit the other post. Or he could make a comment on the other post, but then he's limited by number of words and formatting. Also, the OP is asking for addition information. So what is the standard practice when a duplicate doesn't quite answer your question, or it's poorly phrased? $\endgroup$ – orodbhen Feb 1 '18 at 10:09
  • $\begingroup$ There's a similar question asked before: stats.stackexchange.com/questions/275358/… though it's probably not a duplicate $\endgroup$ – Aksakal Mar 21 '18 at 19:18
  • 1
    $\begingroup$ @orodbhen I think these are great questions to raise on Meta. $\endgroup$ – Sycorax Sep 28 '18 at 22:33

Suppose you want to approximate $f(x)=x^2$ using ReLUs $g(ax+b)$. One approximation might look like $h_1(x)=g(x)+g(-x)=|x|$.


But this isn't a very good approximation. But you can add more terms with different choices of $a$ and $b$ to improve the approximation. One such improvement, in the sense that the error is "small" across a larger interval, is we have $h_2(x)=g(x)+g(-x)+g(2x-2)+g(-2x+2)$, and it gets better.


You can continue this procedure of adding terms to as much complexity as you like.

Notice that, in the first case, the approximation is best for $x\in[-1,1]$, while in the second case, the approximation is best for $x\in[-2,2]$.

enter image description here

x <- seq(-3,3,length.out=1000)
y_true <- x^2
relu <- function(x,a=1,b=0) sapply(x, function(t) max(a*t+b,0))

h1 <- function(x) relu(x)+relu(-x)
    plot(x, h1(x), type="l")
    lines(x, y_true, col="red")

h2 <- function(x) h1(x) + relu(2*(x-1)) + relu(-2*(x+1))
    plot(x, h2(x), type="l")
    lines(x, y_true, col="red")

l2 <- function(y_true,y_hat) 0.5 * (y_true - y_hat)^2

    plot(x, l2(y_true,h1(x)), type="l")
    lines(x, l2(y_true,h2(x)), col="red")
  • $\begingroup$ Sorry if this is dumb question, ReLu = max(0, x), can you please explain why you used ReLu as g(ax+b) $\endgroup$ – tjt Oct 22 '20 at 1:25
  • 1
    $\begingroup$ Define $g(x)=\max\{x,0\}$. Neural networks are parameterized by affine transformations $ax+b$, so if we combine these operations together we get $g(ax+b)$, which is a composition of one layer's parameters and activation functions into a single operation. It's just notationally convenient for this answer. $\endgroup$ – Sycorax Oct 22 '20 at 1:30
  • $\begingroup$ Thanks for the quick reply. Learnt a lot from several of your answers in the forum. They are very clear. $\endgroup$ – tjt Oct 22 '20 at 1:32

Think of it as a piecewise linear function. Any segment of the function that you want to model (assuming it's smooth) looks like a line if you zoom in far enough.


Not the answer you're looking for? Browse other questions tagged or ask your own question.