enter image description here

I get that for z > 0 , the gradient of the ReLU function is 1 , hence the gradient descent algorithm can proceed faster. But what happens if z < 0? How would the gradient descent algorithm proceed then?

  • 3
    $\begingroup$ Gradient descent still works. It does not adjust any input weights on a ReLU neuron with an activation of less than zero. The neurons which contributed to the network output (i.e. those with z > 0) get weight adjustments. If z < 0 on all the training inputs the neuron never contributes to the output and is effectively pruned from the network. $\endgroup$ – Keith Brodie Oct 18 '17 at 19:31

Both back and forward propagation are faster to evaluate for the same reason: relu doesn't need any calculation. The gradient is either 0 on the negative value or 1 for the positive.

I wouldn't say that the gradient descent moves faster due to gradient being 1 on the positive values. It's only the calculations are faster.

  • $\begingroup$ +1, it would be even better with more explanations. $\endgroup$ – hxd1011 Oct 19 '17 at 5:26

The gradient is zero, which means nothing gets backpropagated through them; the precise value fed to this neuron doesn't matter when $z<0$. If a RELU unit is always in the left part of the graph, you get the dying/dead RELU problem; this is one of the reasons that people have looked for alternatives to RELUs.

Sigmoid activations get a similar problem at both ends; even though they're not exactly flat like a RELU, they're close enough once you get to +/- 5 or so.

  • 1
    $\begingroup$ +1, i think this is better than accepted answer. $\endgroup$ – hxd1011 Oct 19 '17 at 5:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.