We use activation functions in neural nets to introduce some non-linearity. Now I understand that Relu is a non-linear function and I had no problems with it.
But today I learned that when the output of relu is 0, it is called a "dead relu" and it kills the gradient. So we initialize relu units with a small bias so that they stay active. I just really couldn't understand how this is a solution. When relu is active it acts as a linear function, it literally outputs the input. So we are using an activation function and we try to make it act linear? How does this work?

  • 1
    $\begingroup$ There are artificial intelligence forum. They’re looking for cheap and effective. This does both. Linear is cheap, it’s derivative cheaper. The relu part of it makes it nonlinear. $\endgroup$ – EngrStudent Dec 18 '20 at 18:44
  • 2
    $\begingroup$ When the neuron is "dead" it means that it's zero on all or almost all input values. When using relu as an activation function it's normal and expected that the output is going to be 0 a lot, as long as it's not 0 all the time the model can still train. $\endgroup$ – Jonny Lomond Dec 18 '20 at 18:51
  • 1
    $\begingroup$ You can't train a network with back-prop if the gradients are consistently zero because the weights and biases will very rarely update. $\endgroup$ – Sycorax Dec 18 '20 at 19:00
  • $\begingroup$ @JonnyLomond so what we are trying to avoid is not having non-zero outputs for for all relu units (i.e. all relus are active) but just to ensure that relus are not outputting zeroes for the totality of our input data? $\endgroup$ – diane Dec 19 '20 at 16:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.