# What are the advantages of ReLU over sigmoid function in deep neural networks?

The state of the art of non-linearity is to use rectified linear units (ReLU) instead of sigmoid function in deep neural network. What are the advantages?

I know that training a network when ReLU is used would be faster, and it is more biological inspired, what are the other advantages? (That is, any disadvantages of using sigmoid)?

• I was under the impression that allowing non-linearity into your network was an advantage. But I don't see that in either answer below... – Monica Heddneck Sep 16 '16 at 3:31
• @MonicaHeddneck both ReLU and sigmoid are nonlinear... – Antoine Jul 6 '17 at 11:24

Two additional major benefits of ReLUs are sparsity and a reduced likelihood of vanishing gradient. But first recall the definition of a ReLU is $h = \max(0, a)$ where $a = Wx + b$.

One major benefit is the reduced likelihood of the gradient to vanish. This arises when $a > 0$. In this regime the gradient has a constant value. In contrast, the gradient of sigmoids becomes increasingly small as the absolute value of x increases. The constant gradient of ReLUs results in faster learning.

The other benefit of ReLUs is sparsity. Sparsity arises when $a \le 0$. The more such units that exist in a layer the more sparse the resulting representation. Sigmoids on the other hand are always likely to generate some non-zero value resulting in dense representations. Sparse representations seem to be more beneficial than dense representations.

• When you say the gradient, you mean with respect to weights or the input x? @DaemonMaker – MAS Jan 30 '15 at 8:10
• With respect to the weights. Gradient-based learning algorithms always taking the gradient with respect to the parameters of the learner, i.e. the weights and biases in a NN. – DaemonMaker Jan 30 '15 at 12:22
• What do you mean by "dense" and "sparse" "representations" ? Query to google "sparse representation neural networks" doesn't seem to come up with anything relevant. – Hi-Angel Feb 10 '17 at 11:52
• sparsity refers to several weights, or coefficients in regression, being equal to zero. – rafaelvalle Mar 2 '17 at 7:01
• "Sparse representations seem to be more beneficial than dense representations." Could you provide a source or explanation? – Rohan Saxena Jan 28 at 20:54

• Sigmoid: not blowing up activation
• Relu : not vanishing gradient
• Relu : More computationally efficient to compute than Sigmoid like functions since Relu just needs to pick max(0,$x$) and not perform expensive exponential operations as in Sigmoids
• Relu : In practice, networks with Relu tend to show better convergence performance than sigmoid. (Krizhevsky et al.)

• Sigmoid: tend to vanish gradient (cause there is a mechanism to reduce the gradient as "$a$" increase, where "$a$" is the input of a sigmoid function. Gradient of Sigmoid: $S'(a)= S(a)(1-S(a))$. When "$a$" grows to infinite large , $S'(a)= S(a)(1-S(a)) = 1\times(1-1)=0$).

• Relu : tend to blow up activation (there is no mechanism to constrain the output of the neuron, as "$a$" itself is the output)

• Relu : Dying Relu problem - if too many activations get below zero then most of the units(neurons) in network with Relu will simply output zero, in other words, die and thereby prohibiting learning.(This can be handled, to some extent, by using Leaky-Relu instead.)
• You might find it helpful that you can use math typesetting with Latex by putting dollar signs around your equations, e.g. $x$ produces $x$. – Silverfish May 7 '16 at 21:02
• Relu: not vanishing gradient. Huh? $\mbox{Relu}(ax+b)=0$ for all $x<-b/a$. – Alex R. Jun 19 at 21:21

Just complementing the other answers:

The other answers are right to point out that the bigger the input (in absolute value) the smaller the gradient of the sigmoid function. But, probably an even more important effect is that the derivative of the sigmoid function is ALWAYS smaller than one. In fact it is at most 0.25!

The down side of this is that if you have many layers, you will multiply these gradients, and the product of many smaller than 1 values goes to zero very quickly.

Since the state of the art of for Deep Learning has shown that more layers helps a lot, then this disadvantage of the Sigmoid function is a game killer. You just can't do Deep Learning with Sigmoid.

On the other hand the gradient of the ReLu function is either $0$ for $a < 0$ or $1$ for $a > 0$. That means that you can put as many layers as you like, because multiplying the gradients will neither vanish nor explode.

• This is the answer I was looking for. When people are talking about "vanishing gradients" one can't stop wondering "ReLu's gradient is exactly 0 for half of its range. Isn't that 'vanishing'". The way you describe the problem by reminding us that gradients are multiplied over many layers, brings much clarity. – Boris Gorelik Jan 3 at 7:31
• @guilherme-de-lazari suggested correction in last line - value of relu is a for a>0 but you are talking about gradient which is 1 for a>0 – saurabh Feb 18 at 7:32
• If this were the main reason, then couldn't we just rescale the sigmoid to 1/(1+exp(-4x))? Then the derivative is at most 1 (or rescale even more, to give us options above and below 1). I suspect this would perform much worse, because rescaling also reduces the area where the derivative is distinguishable from 0. But I'm not sure this answer tells the full story. – Peter Feb 21 at 13:09
• This answer is nonsense. The derivative of a sigmoid with constant parameter 1 is less than 1. But more generally it's $1/(1+\exp(-ax))$, which can have an arbitrarily large derivative (just take $a$ to be really large, so the sigmoid rapidly goes from 0 to 1). – Alex R. Jun 19 at 21:12
• Also you CAN do deep learning with sigmoids, you just need to normalize the inputs, for example via Batch Normalization. This will centralize your inputs to avoid saturating the sigmoid. In the original paper on Batch Normalization, the sigmoid activation neural network does nearly on par with ReLus: arxiv.org/pdf/1502.03167.pdf – Alex R. Jun 19 at 21:15

An advantage to ReLU other than avoiding vanishing gradients problem is that it has much lower run time. max(0,a) runs much faster than any sigmoid function (logistic function for example = 1/(1+e^(-a)) which uses an exponent which is computational slow when done often). This is true for both feed forward and back propagation as the gradient of ReLU (if a<0, =0 else =1) is also very easy to compute compared to sigmoid (for logistic curve=e^a/((1+e^a)^2)).

Although ReLU does have the disadvantage of dying cells which limits the capacity of the network. To overcome this just use a variant of ReLU such as leaky ReLU, ELU,etc if you notice the problem described above.

• +1. This is one of the only correct answers here. You can also use batch normalization to centralize inputs to counteract dead neurons. – Alex R. Jun 19 at 21:08

An extra piece of answer to complete on the Sparse vs Dense performance debate.

Don't think about NN anymore, just think about linear algebra and matrix operations, because forward and backward propagations are a series of matrix operations.

Now remember that there exist a lot of optimized operator to apply to sparse matrix and so optimizing those operations in our network could dramatically improve the performance of the algorithm.

I hope that could help some of you guys...