Many people claim to use ReLU to solve vanishing gradient problem, but wouldn't dying ReLU be a more serious problem? And someone also claims that ELU performs better, but wouldn't ELU also suffer from vanishing gradient? PReLU seems to avoid the problem, but it is not very popular. What's the catch?
$\begingroup$
$\endgroup$
3
-
1$\begingroup$ I've had fine luck with leaky ReLU's. I think that it's all just so new, and that practice is so far ahead of theory, that nobody really knows. $\endgroup$– generic_userMay 12, 2017 at 14:59
-
2$\begingroup$ I recommend watching this talk by Moritz Hardt (youtube.com/watch?v=l1YxQ1Od1Y0) where he explains that ReLU in fact does not solve the vanishing gradient problem. $\endgroup$– Jan KukackaDec 25, 2017 at 20:20
-
$\begingroup$ @JanKukacka Here is exactly where Moritz Hardt starts explaining about that. Though it seems to me that Moritz mostly explain about dying ReLU, and then kind of claims, like wikipedia, that dying ReLU is a form of vanishing gradient. So I think that it is mainly a disagreement about terminology: I think everyone agrees that dying ReLU can happen, but Moritz calls its consequences "vanishing gradient", while others call the same consequences "sparse network". $\endgroup$– Oren MilmanSep 27, 2018 at 17:19
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
0
ELU and ReLU both have zero or vanishing gradient "on the left". This is still a marked departure from $\tanh$ or logistic units, because those functions are bounded above and below; for ELU and ReLU units, the gradient updates will be larger "on the right". As a demonstration, work out the derivatives for each and note that the logistic and $\tanh$ units usually have smaller gradients for inputs in some interval around 0 such as $ [-2,2]$ than ELU and PReLU; $\tanh$ only attains a gradient of 1 at zero, and the logistic unit not at all! On the other hand, ReLU/ELU/PReLU have gradient 1 for all positive inputs.
On the other hand, you're correct that PReLUs avoid having a zero gradient everywhere. I'm not aware of a study exhaustively comparing ELU, ReLU and PReLU units. There's still a long way to go between these practical innovations in neural networks and a theoretical understanding of why they work well.
answered Jun 10, 2018 at 14:22