Isn't ReLU just a linear function? I am doing Andew Ng's Deep Learning course and he says that ReLU is better than Sigmoid, but it makes no sense to me at all.
The biggest advantage of activation functions are too get a non-linear function that means that cascading is non-linear and when stacking layers, we can learn non-linear functions.
But I can't understand why ReLU is anybetter since it is still a straight line. Just two straight lines. The function is discontinous at zero too from what I understand?
It just makes no sense to me why ReLU would be a good function.
Edit: Nothing about ReLU makes sense to me. It is two straight lines made worse by one value being forced to zero and I cannot understand why we'd want to force $g(z)=0  \forall z<0$. I can't see how this is any better than $g(z)=z$
 A: As can be seen in the pytorch documentation for the ReLU, this is not a straight line, and not discontinuous at zero, two important points not to miss:

First, if it would be linear, then the cascade of linear operations (such as a convolution) with it would just be yet another linear operator, such that you could as well collapse this cascade into a shallow one-layer transform. It is essential to understand that the goal is to create the most diverse "space" of functions - and that non-linearities such as the ReLU are essential for this. (Figure 1 of this review may help understand why non-linearities are essential.)
Second, it is non discontinuous at zero - this would generate problems for establishing the gradients which are essential to perform backpropagation. Instead, it is continuous at zero and its derivate (which is discontinuous at zero) is most easily computable as zero for negative values and one for positive.
Hope this helps!
Edit: Following other answers to similar questions, another reason for which the ReLU non-linearity is popular is the fact that it helps overcome the vanishing gradient problem. Indeed, when using a smooth function such as the sigmoid, the value of the gradient computed in back-propagation (from the last layer to the first) could become very small for deep networks - slowing down learning in the first layers. The ReLU (among other mechanisms such as batch normalization) can help overcome this problem due to its shape.
A: You should not confuse yourself in a way that you transform your values to a ReLu Function. You only use ReLu to, well, create several lines which will be "added together" so you can get several different kind of fitting functions. And it is much faster than Sigmoid. See here for using ReLu:
ReLu explained by Josh Starmer on StatQuest
https://www.youtube.com/watch?v=68BZ5f7P94E
