0
$\begingroup$

As far as I understand, Signoid function is used for mapping the outputs of neural network to the values between 0 and 1. Why is using rectified linear unit(ReLU) as activation function in deep neural networks, works faster? Can you please explain the mathematical concept behind it?

Improve this question
$\endgroup$
3
  • $\begingroup$ "Faster" in what sense? If you ask about computation, max in ReLU just compares two numbers, while exp does a number of different computations. $\endgroup$
    – Tim ♦
    Jan 2, 2019 at 14:28
  • $\begingroup$ I am now learning from Andrew Ng's deep learning specialization and I heard that using ReLU instead of Signoid was one of the great breakthroughs for deep learning because it helps make the gradient descent works much faster for large deep neural networks My understanding of activation functions is that it maps the outputs of a neural net to a range of values so I am kinda lost with the concept of ReLU making the gradient descent process faster. $\endgroup$
    – Htut Lin Aung
    Jan 2, 2019 at 14:37
  • 1
    $\begingroup$ Long story short, the answer is "steeper gradients" -- this is described precisely in my answer here: stats.stackexchange.com/questions/226923/… The key detail is that the network trains faster, not that the gradient descent algorithm is any different. $\endgroup$
    – Sycorax ♦
    Jan 2, 2019 at 15:11

0

Reset to default

Browse other questions tagged or ask your own question.