Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
The state of the art of non-linearity is to use rectified linear units (ReLU) instead of sigmoid function in deep neural network. What are the advantages?
I know that training a network when ReLU is used would be faster, and it is more biological inspired, what are the other advantages? (That is, any disadvantages of using sigmoid)?
Two additional major benefits of ReLUs are sparsity and a reduced likelihood of vanishing gradient. But first recall the definition of a ReLU is $h = \max(0, a)$ where $a = Wx + b$.
One major benefit is the reduced likelihood of the gradient to vanish. This arises when $a > 0$. In this regime the gradient has a constant value. In contrast, the gradient of sigmoids becomes increasingly small as the absolute value of x increases. The constant gradient of ReLUs results in faster learning.
The other benefit of ReLUs is sparsity. Sparsity arises when $a \le 0$. The more such units that exist in a layer the more sparse the resulting representation. Sigmoids on the other hand are always likely to generate some non-zero value resulting in dense representations. Sparse representations seem to be more beneficial than dense representations.
Advantage:
Disadvantage:
Sigmoid: tend to vanish gradient (cause there is a mechanism to reduce the gradient as "$a$" increase, where "$a$" is the input of a sigmoid function. Gradient of Sigmoid: $S'(a)= S(a)(1-S(a))$. When "$a$" grows to infinite large , $S'(a)= S(a)(1-S(a)) = 1\times(1-1)=0$).
Relu : tend to blow up activation (there is no mechanism to constrain the output of the neuron, as "$a$" itself is the output)
Just complementing the other answers:
Vanishing Gradients
The other answers are right to point out that the bigger the input (in absolute value) the smaller the gradient of the sigmoid function. But, probably an even more important effect is that the derivative of the sigmoid function is ALWAYS smaller than one. In fact it is at most 0.25!
The down side of this is that if you have many layers, you will multiply these gradients, and the product of many smaller than 1 values goes to zero very quickly.
Since the state of the art of for Deep Learning has shown that more layers helps a lot, then this disadvantage of the Sigmoid function is a game killer. You just can't do Deep Learning with Sigmoid.
On the other hand the gradient of the ReLu function is either $0$ for $a < 0$ or $1$ for $a > 0$. That means that you can put as many layers as you like, because multiplying the gradients will neither vanish nor explode.
An advantage to ReLU other than avoiding vanishing gradients problem is that it has much lower run time. max(0,a) runs much faster than any sigmoid function (logistic function for example = 1/(1+e^(-a)) which uses an exponent which is computational slow when done often). This is true for both feed forward and back propagation as the gradient of ReLU (if a<0, =0 else =1) is also very easy to compute compared to sigmoid (for logistic curve=e^a/((1+e^a)^2)).
Although ReLU does have the disadvantage of dying cells which limits the capacity of the network. To overcome this just use a variant of ReLU such as leaky ReLU, ELU,etc if you notice the problem described above.
The main reason why ReLu is used is because it is simple, fast, and empirically it seems to work well.
Empirically, early papers observed that training a deep network with ReLu tended to converge much more quickly and reliably than training a deep network with sigmoid activation. In the early days, people were able to train deep networks with ReLu but training deep networks with sigmoid flat-out failed. There are many hypotheses that have attempted to explain why this could be.
First, with a standard sigmoid activation, the gradient of the sigmoid is typically some fraction between 0 and 1; if you have many layers, these multiply, and might give an overall gradient that is exponentially small, so each step of gradient descent will make only a tiny change to the weights, leading to slow convergence (the vanishing gradient problem). In contrast, with ReLu activation, the gradient of the ReLu is either 0 or 1, so after many layers often the gradient will include the product of a bunch of 1's, and thus the overall gradient is not too small or not too large. But this story might be too simplistic, because it doesn't take into account the way that we multiply by the weights and add up internal activations.
Second, with sigmoid activation, the gradient goes to zero if the input is very large or very small. When the gradient goes to zero, gradient descent tends to have very slow convergence. In contrast, with ReLu activation, the gradient goes to zero if the input is negative but not if the input is large, so it might have only "half" of the problems of sigmoid. But this seems a bit naive too as it is clear that negative values still give a zero gradient.
Since then, we've accumulated more experience and more tricks that can be used to train neural networks. For instance, batch normalization is very helpful. When you add in those tricks, the comparison becomes less clear. It is possible to successfully train a deep network with either sigmoid or ReLu, if you apply the right set of tricks.
I suspect that ultimately there are several reasons for widespread use of ReLu today:
Historical accident: we discovered ReLu in the early days before we knew about those tricks, so in the early days ReLu was the only choice that worked, and everyone had to use it. And now that everyone uses it, it is a safe choice and people keep using it.
Efficiency: ReLu is faster to compute than the sigmoid function, and its derivative is faster to compute. This makes a significant difference to training and inference time for neural networks: only a constant factor, but constants can matter.
Simplicity: ReLu is simple.
Fragility: empirically, ReLu seems to be a bit more forgiving (in terms of the tricks needed to make the network train successfully), whereas sigmoid is more fiddly (to train a deep network, you need more tricks, and it's more fragile).
Good enough: empirically, in many domains, other activation functions are no better than ReLu, or if they are better, are better by only a tiny amount. So, if ReLu is simple, fast, and about as good as anything else in most settings, it makes a reasonable default.
Main benefit is that the derivative of ReLu is either 0 or 1, so multiplying by it won't cause weights that are further away from the end result of the loss function to suffer from the vanishing gradient problem:
ReLu does not have the vanishing gradient problem. Vanishing gradients lead to very small changes in the weights proportional to the partial derivative of the error function. The gradient is multiplied n times in back propagation to get the gradients of lower layers. The effect of multiplying the gradient n times makes the gradient to be even smaller for lower layers, leading to a very small change or even no change in the weights of lower layers. Therefore, the deeper the network, the more the effect of vanishing gradients. This makes learning per iteration slower when activation functions that suffer from vanishing gradients is used e.g Sigmoid and tanh functions. Kindly refer here
ReLU function is not computationally heavy to compute compared to sigmoid function. This is well covered above.
