# How does the ReLU function work for z < 0?

I get that for z > 0 , the gradient of the ReLU function is 1 , hence the gradient descent algorithm can proceed faster. But what happens if z < 0? How would the gradient descent algorithm proceed then?

• Gradient descent still works. It does not adjust any input weights on a ReLU neuron with an activation of less than zero. The neurons which contributed to the network output (i.e. those with z > 0) get weight adjustments. If z < 0 on all the training inputs the neuron never contributes to the output and is effectively pruned from the network. – Keith Brodie Oct 18 '17 at 19:31

## 2 Answers

Both back and forward propagation are faster to evaluate for the same reason: relu doesn't need any calculation. The gradient is either 0 on the negative value or 1 for the positive.

I wouldn't say that the gradient descent moves faster due to gradient being 1 on the positive values. It's only the calculations are faster.

• +1, it would be even better with more explanations. – hxd1011 Oct 19 '17 at 5:26

The gradient is zero, which means nothing gets backpropagated through them; the precise value fed to this neuron doesn't matter when $z<0$. If a RELU unit is always in the left part of the graph, you get the dying/dead RELU problem; this is one of the reasons that people have looked for alternatives to RELUs.

Sigmoid activations get a similar problem at both ends; even though they're not exactly flat like a RELU, they're close enough once you get to +/- 5 or so.

• +1, i think this is better than accepted answer. – hxd1011 Oct 19 '17 at 5:26