I am doing Andew Ng's Deep Learning course and he says that ReLU is better than Sigmoid, but it makes no sense to me at all.
The biggest advantage of activation functions are too get a non-linear function that means that cascading is non-linear and when stacking layers, we can learn non-linear functions.
But I can't understand why ReLU is anybetter since it is still a straight line. Just two straight lines. The function is discontinous at zero too from what I understand?
It just makes no sense to me why ReLU would be a good function.
Edit: Nothing about ReLU makes sense to me. It is two straight lines made worse by one value being forced to zero and I cannot understand why we'd want to force $g(z)=0 \forall z<0$. I can't see how this is any better than $g(z)=z$