# How does the ReLU function work for z < 0?

I get that for z > 0 , the gradient of the ReLU function is 1 , hence the gradient descent algorithm can proceed faster. But what happens if z < 0? How would the gradient descent algorithm proceed then?

• Gradient descent still works. It does not adjust any input weights on a ReLU neuron with an activation of less than zero. The neurons which contributed to the network output (i.e. those with z > 0) get weight adjustments. If z < 0 on all the training inputs the neuron never contributes to the output and is effectively pruned from the network. – Keith Brodie Oct 18 '17 at 19:31

The gradient is zero, which means nothing gets backpropagated through them; the precise value fed to this neuron doesn't matter when $z<0$. If a RELU unit is always in the left part of the graph, you get the dying/dead RELU problem; this is one of the reasons that people have looked for alternatives to RELUs.