3
$\begingroup$

I have already gone through the post and this post, but they didn't clear my doubt. Let us say if I have a deep neural network like (having more layers about 50):

Enter image description here

Now, my question is:

If I'm using an activation function as the ReLU, the gradient will be 1 for all values of x>0 and 0 for for all values of x<0. So, where do the notion of vanishing gradients occur for a ReLU? I thought a ReLu was known for solving the vanishing gradient problem.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
8
$\begingroup$

ReLUs don't suffer from vanishing gradients, but they have their own issues. The main issue for ReLU is the dead-ReLU phenomenon, i.e. a neuron whose activation becomes zero for all samples due to ReLU. This neuron is irrecoverable, because gradients through it are always zero.

Leaky ReLU and other activation functions were created to deal with this issue.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Then why Resnet (residual networks) comes into a picture for deep neural networks if we can improve the vanishing gradient with leaky ReLu itself? $\endgroup$
    – Bits
    Jul 26, 2021 at 19:05
  • 3
    $\begingroup$ @Bits skip-layer connections have been around for a very long time, solving the vanishing gradients problem isn't the only reason. Sometimes you can get a more compact solution to the problem if you have skip layer connections as well (for instance many problems have linear components, which are most easily implemented by direct connections from input to output layers). Also engineers like ""belt and braces" solutions to problems, especially when they are relatively cheap to implement. $\endgroup$
    – Dikran Marsupial
    Jul 26, 2021 at 19:11
  • 1
    $\begingroup$ @Bits the raison d'etre for ResNets is that, with residual connections, you can implement "residual learning", in the sense that you learn increments to be added to previous layers. These, in practice, should be easier to learn than the actual function. ResNet wasn't created to solve vanishing gradients. $\endgroup$
    – Firebug
    Jul 26, 2021 at 19:16
  • $\begingroup$ @Firebug So, can I say that if leaky Relu used as activation function in neural network than vanishing gradient will never occur in such networks no matter how deep neural network is? $\endgroup$
    – Bits
    Jul 27, 2021 at 5:34
  • $\begingroup$ @Bits theroretically, yes. In practice, not quite. There's a limit involved due to floating point precision. $\endgroup$
    – Firebug
    Jul 27, 2021 at 11:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.