As far as I understand, Signoid function is used for mapping the outputs of neural network to the values between 0 and 1. Why is using rectified linear unit(ReLU) as activation function in deep neural networks, works faster? Can you please explain the mathematical concept behind it?


marked as duplicate by Jakub Bartczuk, Sycorax, Ferdi, kjetil b halvorsen, jbowman Jan 2 at 17:14

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ "Faster" in what sense? If you ask about computation, max in ReLU just compares two numbers, while exp does a number of different computations. $\endgroup$ – Tim Jan 2 at 14:28
  • $\begingroup$ I am now learning from Andrew Ng's deep learning specialization and I heard that using ReLU instead of Signoid was one of the great breakthroughs for deep learning because it helps make the gradient descent works much faster for large deep neural networks My understanding of activation functions is that it maps the outputs of a neural net to a range of values so I am kinda lost with the concept of ReLU making the gradient descent process faster. $\endgroup$ – Htut Lin Aung Jan 2 at 14:37
  • 1
    $\begingroup$ Long story short, the answer is "steeper gradients" -- this is described precisely in my answer here: stats.stackexchange.com/questions/226923/… The key detail is that the network trains faster, not that the gradient descent algorithm is any different. $\endgroup$ – Sycorax Jan 2 at 15:11