3
$\begingroup$

In this link about different neuron types, there is an introduction on the disadvantage of ReLU,

(-) Unfortunately, ReLU units can be fragile during training and can "die". For example, a large gradient flowing through a ReLU neuron could cause the weights to update in such a way that the neuron will never activate on any datapoint again. If this happens, then the gradient flowing through the unit will forever be zero from that point on. That is, the ReLU units can irreversibly die during training since they can get knocked off the data manifold. For example, you may find that as much as 40% of your network can be "dead" (i.e. neurons that never activate across the entire training dataset) if the learning rate is set too high. With a proper setting of the learning rate this is less frequently an issue.

I am not clear why large gradient flowing through a ReLU neuron could cause the neuron to die.

Please see the following ReLU picture for reference.

enter image description here

Improve this question
$\endgroup$

1 Answer 1

Reset to default
4
$\begingroup$

A big update can move the coefficient and bias by such a large amount that for all incoming values to the neuron, the ReLU returns 0 for all inputs. Thus the gradient becomes 0 for all subsequent updates.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I added a picture in my original post. Looks like when incoming values become quite big, then the returned value will increase linearly as well, instead of approaching zero. $\endgroup$
    – user3269
    Sep 28, 2016 at 18:23
  • $\begingroup$ Incoming function values are different from gradient updates. $\endgroup$
    – Sycorax ♦
    Sep 28, 2016 at 18:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.