# Vanishing gradient vs. dying ReLU? [duplicate]

Many people claim to use ReLU to solve vanishing gradient problem, but wouldn't dying ReLU be a more serious problem? And someone also claims that ELU performs better, but wouldn't ELU also suffer from vanishing gradient? PReLU seems to avoid the problem, but it is not very popular. What's the catch?

• I've had fine luck with leaky ReLU's. I think that it's all just so new, and that practice is so far ahead of theory, that nobody really knows. May 12, 2017 at 14:59
• I recommend watching this talk by Moritz Hardt (youtube.com/watch?v=l1YxQ1Od1Y0) where he explains that ReLU in fact does not solve the vanishing gradient problem. Dec 25, 2017 at 20:20
• @JanKukacka Here is exactly where Moritz Hardt starts explaining about that. Though it seems to me that Moritz mostly explain about dying ReLU, and then kind of claims, like wikipedia, that dying ReLU is a form of vanishing gradient. So I think that it is mainly a disagreement about terminology: I think everyone agrees that dying ReLU can happen, but Moritz calls its consequences "vanishing gradient", while others call the same consequences "sparse network". Sep 27, 2018 at 17:19

ELU and ReLU both have zero or vanishing gradient "on the left". This is still a marked departure from $\tanh$ or logistic units, because those functions are bounded above and below; for ELU and ReLU units, the gradient updates will be larger "on the right". As a demonstration, work out the derivatives for each and note that the logistic and $\tanh$ units usually have smaller gradients for inputs in some interval around 0 such as $[-2,2]$ than ELU and PReLU; $\tanh$ only attains a gradient of 1 at zero, and the logistic unit not at all! On the other hand, ReLU/ELU/PReLU have gradient 1 for all positive inputs.