i read that sigmoid function will kill the gradient as a result the network will not learn
and i read that in ReLU function a large gradient flowing through a ReLU neuron could cause the weights to update in such a way that the neuron will never activate
and that may lead “dead” neurons.
so, are they the same?
They both get zero or very small gradients so they can barely get trained.
The difference is, the activation values of “dead” ReLU neurons are almost always zero, whereas the activation values of saturated sigmoid neurons are close to 0 or 1.
Both of them will have very small gradients, hence both act as a showstopper to learning.
Difference is that the likelihood of dead Relu neurons is much less as compared to saturated sigmoids
The gradient of a sigmoid is: S′(a)=S(a)(1−S(a)) When we start learning useful features in the later layers, the activations S(a) are high and thus the gradient or the learning of the previous layers starts reducing.
Not matter how careful you are with network parameters you will encounter saturated sigmoids and vanishing gradients in your network.
For RelU however the gradients are a constant no matter how good the activation/features are in your last layers, thus the previous layers continue to learn. Unless you really mess up the learning rate, or the weight decay terms, or your bias which forces the RelU's to operate in negative regime for all inputs, it is hard to get a lot of "dead" neurons as compared to saturated sigmoids.
