In this link about different neuron types, there is an introduction on the disadvantage of ReLU,

(-) Unfortunately, ReLU units can be fragile during training and can "die". For example, a large gradient flowing through a ReLU neuron could cause the weights to update in such a way that the neuron will never activate on any datapoint again. If this happens, then the gradient flowing through the unit will forever be zero from that point on. That is, the ReLU units can irreversibly die during training since they can get knocked off the data manifold. For example, you may find that as much as 40% of your network can be "dead" (i.e. neurons that never activate across the entire training dataset) if the learning rate is set too high. With a proper setting of the learning rate this is less frequently an issue.

I am not clear why large gradient flowing through a ReLU neuron could cause the neuron to die.

Please see the following ReLU picture for reference.

enter image description here


A big update can move the coefficient and bias by such a large amount that for all incoming values to the neuron, the ReLU returns 0 for all inputs. Thus the gradient becomes 0 for all subsequent updates.

| cite | improve this answer | |
  • $\begingroup$ I added a picture in my original post. Looks like when incoming values become quite big, then the returned value will increase linearly as well, instead of approaching zero. $\endgroup$ – user3269 Sep 28 '16 at 18:23
  • $\begingroup$ Incoming function values are different from gradient updates. $\endgroup$ – Sycorax Sep 28 '16 at 18:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.