1
$\begingroup$

I am doing Andew Ng's Deep Learning course and he says that ReLU is better than Sigmoid, but it makes no sense to me at all.

The biggest advantage of activation functions are too get a non-linear function that means that cascading is non-linear and when stacking layers, we can learn non-linear functions.

But I can't understand why ReLU is anybetter since it is still a straight line. Just two straight lines. The function is discontinous at zero too from what I understand?

It just makes no sense to me why ReLU would be a good function.

Edit: Nothing about ReLU makes sense to me. It is two straight lines made worse by one value being forced to zero and I cannot understand why we'd want to force $g(z)=0 \forall z<0$. I can't see how this is any better than $g(z)=z$

Improve this question
$\endgroup$
1
  • $\begingroup$ HI @mr-johnny-doe ! Do not hesitate to comment our answers if you need more clarifications... $\endgroup$
    – meduz
    Commented Mar 17, 2021 at 11:51

2 Answers 2

Reset to default
3
$\begingroup$

As can be seen in the pytorch documentation for the ReLU, this is not a straight line, and not discontinuous at zero, two important points not to miss:

pytorch documentation for the ReLU

First, if it would be linear, then the cascade of linear operations (such as a convolution) with it would just be yet another linear operator, such that you could as well collapse this cascade into a shallow one-layer transform. It is essential to understand that the goal is to create the most diverse "space" of functions - and that non-linearities such as the ReLU are essential for this. (Figure 1 of this review may help understand why non-linearities are essential.)

Second, it is non discontinuous at zero - this would generate problems for establishing the gradients which are essential to perform backpropagation. Instead, it is continuous at zero and its derivate (which is discontinuous at zero) is most easily computable as zero for negative values and one for positive.

Hope this helps!

Edit: Following other answers to similar questions, another reason for which the ReLU non-linearity is popular is the fact that it helps overcome the vanishing gradient problem. Indeed, when using a smooth function such as the sigmoid, the value of the gradient computed in back-propagation (from the last layer to the first) could become very small for deep networks - slowing down learning in the first layers. The ReLU (among other mechanisms such as batch normalization) can help overcome this problem due to its shape.

Improve this answer
$\endgroup$
11
  • $\begingroup$ Because of "Second" I could write my post a little bit shorter ;D $\endgroup$
    – Patrick Bormann
    Commented Mar 10, 2021 at 13:26
  • $\begingroup$ I stumbled on this question while writing another one ;-) $\endgroup$
    – meduz
    Commented Mar 10, 2021 at 13:33
  • $\begingroup$ I get that it may not be a straight line. But I can't associate any intuition with ReLU. max(0,Z) just feels... wrong? Like something about it rubs me as being bad math. A good chunk of values remain untouched while another chunk of values are saturated to zero. It is very prone to exploding gradients too since the values don't saturate when Z>0. $\endgroup$
    – Mr. Johnny Doe
    Commented Mar 10, 2021 at 13:35
  • $\begingroup$ The only good feature about ReLU from what I can see is that at least its slope is very easy to calculate. $\endgroup$
    – Mr. Johnny Doe
    Commented Mar 10, 2021 at 13:36
  • $\begingroup$ @Mr.JohnnyDoe Don't forget about the bias: You don't necessarily throw away a lot at all. By changing the bias, the threshold where values are set to zero also changes. $\endgroup$
    – Frans Rodenburg
    Commented Mar 10, 2021 at 13:39
1
$\begingroup$

You should not confuse yourself in a way that you transform your values to a ReLu Function. You only use ReLu to, well, create several lines which will be "added together" so you can get several different kind of fitting functions. And it is much faster than Sigmoid. See here for using ReLu:

ReLu explained by Josh Starmer on StatQuest https://www.youtube.com/watch?v=68BZ5f7P94E

Improve this answer
$\endgroup$
1
  • $\begingroup$ I love John, but ReLU is the least intuitive function I have ever seen in math and machine learning. It treats values before zero and after zero very differently. Sigmoid makes sense, 1 and 0 relate to class labels. Tanh relates to how we normalise data. But ReLU just doesn't make sense to me. $\endgroup$
    – Mr. Johnny Doe
    Commented Mar 10, 2021 at 13:34

Not the answer you're looking for? Browse other questions tagged or ask your own question.