What are the advantages of ReLU over sigmoid function in deep neural networks?

Question

The state of the art of non-linearity is to use rectified linear units (ReLU) instead of sigmoid function in deep neural network. What are the advantages?

I know that training a network when ReLU is used would be faster, and it is more biological inspired, what are the other advantages? (That is, any disadvantages of using sigmoid)?

Answer 1

Two additional major benefits of ReLUs are sparsity and a reduced likelihood of vanishing gradient. But first recall the definition of a ReLU is $h = \max(0, a)$ where $a = Wx + b$.

One major benefit is the reduced likelihood of the gradient to vanish. This arises when $a > 0$. In this regime the gradient has a constant value. In contrast, the gradient of sigmoids becomes increasingly small as the absolute value of x increases. The constant gradient of ReLUs results in faster learning.

The other benefit of ReLUs is sparsity. Sparsity arises when $a \le 0$. The more such units that exist in a layer the more sparse the resulting representation. Sigmoids on the other hand are always likely to generate some non-zero value resulting in dense representations. Sparse representations seem to be more beneficial than dense representations.

Answer 2

Advantage:

• Sigmoid: not blowing up activation
• Relu : not vanishing gradient
• Relu : More computationally efficient to compute than Sigmoid like functions since Relu just needs to pick max(0,$x$) and not perform expensive exponential operations as in Sigmoids
• Relu : In practice, networks with Relu tend to show better convergence performance than sigmoid. (Krizhevsky et al.)

Disadvantage:

• Sigmoid: tend to vanish gradient (cause there is a mechanism to reduce the gradient as "$a$" increase, where "$a$" is the input of a sigmoid function. Gradient of Sigmoid: $S'(a)= S(a)(1-S(a))$. When "$a$" grows to infinite large , $S'(a)= S(a)(1-S(a)) = 1\times(1-1)=0$).

• Relu : tend to blow up activation (there is no mechanism to constrain the output of the neuron, as "$a$" itself is the output)

• Relu : Dying Relu problem - if too many activations get below zero then most of the units(neurons) in network with Relu will simply output zero, in other words, die and thereby prohibiting learning.(This can be handled, to some extent, by using Leaky-Relu instead.)

Answer 3

Just complementing the other answers:

Vanishing Gradients

The other answers are right to point out that the bigger the input (in absolute value) the smaller the gradient of the sigmoid function. But, probably an even more important effect is that the derivative of the sigmoid function is ALWAYS smaller than one. In fact it is at most 0.25!

The down side of this is that if you have many layers, you will multiply these gradients, and the product of many smaller than 1 values goes to zero very quickly.

Since the state of the art of for Deep Learning has shown that more layers helps a lot, then this disadvantage of the Sigmoid function is a game killer. You just can't do Deep Learning with Sigmoid.

On the other hand the gradient of the ReLu function is either $0$ for $a < 0$ or $1$ for $a > 0$. That means that you can put as many layers as you like, because multiplying the gradients will neither vanish nor explode.

Answer 4

An advantage to ReLU other than avoiding vanishing gradients problem is that it has much lower run time. max(0,a) runs much faster than any sigmoid function (logistic function for example = 1/(1+e^(-a)) which uses an exponent which is computational slow when done often). This is true for both feed forward and back propagation as the gradient of ReLU (if a<0, =0 else =1) is also very easy to compute compared to sigmoid (for logistic curve=e^a/((1+e^a)^2)).

Although ReLU does have the disadvantage of dying cells which limits the capacity of the network. To overcome this just use a variant of ReLU such as leaky ReLU, ELU,etc if you notice the problem described above.

Answer 5

The main reason why ReLu is used is because it is simple, fast, and empirically it seems to work well.

Empirically, early papers observed that training a deep network with ReLu tended to converge much more quickly and reliably than training a deep network with sigmoid activation. In the early days, people were able to train deep networks with ReLu but training deep networks with sigmoid flat-out failed. There are many hypotheses that have attempted to explain why this could be.

• First, with a standard sigmoid activation, the gradient of the sigmoid is typically some fraction between 0 and 1; if you have many layers, these multiply, and might give an overall gradient that is exponentially small, so each step of gradient descent will make only a tiny change to the weights, leading to slow convergence (the vanishing gradient problem). In contrast, with ReLu activation, the gradient of the ReLu is either 0 or 1, so after many layers often the gradient will include the product of a bunch of 1's, and thus the overall gradient is not too small or not too large. But this story might be too simplistic, because it doesn't take into account the way that we multiply by the weights and add up internal activations.

• Second, with sigmoid activation, the gradient goes to zero if the input is very large or very small. When the gradient goes to zero, gradient descent tends to have very slow convergence. In contrast, with ReLu activation, the gradient goes to zero if the input is negative but not if the input is large, so it might have only "half" of the problems of sigmoid. But this seems a bit naive too as it is clear that negative values still give a zero gradient.

Since then, we've accumulated more experience and more tricks that can be used to train neural networks. For instance, batch normalization is very helpful. When you add in those tricks, the comparison becomes less clear. It is possible to successfully train a deep network with either sigmoid or ReLu, if you apply the right set of tricks.

I suspect that ultimately there are several reasons for widespread use of ReLu today:

1. Historical accident: we discovered ReLu in the early days before we knew about those tricks, so in the early days ReLu was the only choice that worked, and everyone had to use it. And now that everyone uses it, it is a safe choice and people keep using it.

2. Efficiency: ReLu is faster to compute than the sigmoid function, and its derivative is faster to compute. This makes a significant difference to training and inference time for neural networks: only a constant factor, but constants can matter.

3. Simplicity: ReLu is simple.

4. Fragility: empirically, ReLu seems to be a bit more forgiving (in terms of the tricks needed to make the network train successfully), whereas sigmoid is more fiddly (to train a deep network, you need more tricks, and it's more fragile).

5. Good enough: empirically, in many domains, other activation functions are no better than ReLu, or if they are better, are better by only a tiny amount. So, if ReLu is simple, fast, and about as good as anything else in most settings, it makes a reasonable default.

Answer 6

Main benefit is that the derivative of ReLu is either 0 or 1, so multiplying by it won't cause weights that are further away from the end result of the loss function to suffer from the vanishing gradient problem:

Answer 7

1. ReLu does not have the vanishing gradient problem. Vanishing gradients lead to very small changes in the weights proportional to the partial derivative of the error function. The gradient is multiplied n times in back propagation to get the gradients of lower layers. The effect of multiplying the gradient n times makes the gradient to be even smaller for lower layers, leading to a very small change or even no change in the weights of lower layers. Therefore, the deeper the network, the more the effect of vanishing gradients. This makes learning per iteration slower when activation functions that suffer from vanishing gradients is used e.g Sigmoid and tanh functions. Kindly refer here

2. ReLU function is not computationally heavy to compute compared to sigmoid function. This is well covered above.

Answer 8

I read all the answers and still feel that I need to write a new one.

Let us consider a linear activation function g(z)=z, which is different from Relu(z) only in the region z<0. If all activation functions used in a network is g(z), then the network is equivalent to a simple single layer linear network, which we know is not useful in learning complicate patterns. We need to introduce nonlinearity into the network. So the interesting part of Relu(z) is actually its combination of this linear part with another linear part that has a different slope (specifically, zero in Relu). This introduces a nonlinearity we need, which seems to be the most simple nonlinearity that one can think of.

However simplicity itself does not imply superiorness over complexity in terms of its practical use.

Then why is this simple nonlinearity more powerful than the sigmoid function?

Both relu and sigmoid have regions of zero derivative. Other answers have claimed that relu has a reduced chance of encountering the vanishing gradient problem based on the facts that (1) its zero derivative region is narrower than sigmoid and (2) relu's derivative for z>0 is equal to one, which is not damped or enhanced when multiplied.

Other possible reasons for the advantage of relu over sigmoid may be that (1) Relu has larger possible range than that of the sigmoid function for z>0. (2) The exact zero values of relu for z<0 introduce sparsity effect in the network, which forces the network to learn more robust features. If this is true, something like leaky Relu, which is claimed as an improvement over relu, may be actually damaging the efficacy of Relu.

Some people consider relu very strange at first glance. It turns out that the adoption of relu is a natural choice if we consider that (1) sigmoid is a modified version of the step function (g=0 for z<0, and g=1 for z>0) to make it continuous near zero; (2) another imaginable modified version of the step function would be replacing g=1 in z>0 by g=z, which is relu.

Answer 9

<ReLU>

Pros:

• It mitigates Vanishing Gradient Problem.
• It's not computationally expensive.

Cons:

• It causes Dying ReLU Problem.
• It's non-differentiable at x=0.

<Sigmoid>

Pros:

• It normalises input values so the convergence is more stable than ReLU.
• It mitigates Exploding Gradient Problem.
• It avoids Dying ReLU Problem.
• It's continuously differentiable.

Cons:

• It causes Vanishing Gradient Problem.
• It's computationally expensive because of exponential operation.