Skip to main content
Added a few more Advantages and Disadvantages for Relu-Sigmoid comparison. a brief explanations for each and required references provided.
Source Link

Advantage:

  • Sigmoid: not blowing up activation
  • Relu : not vanishing gradient
  • Relu : More computationally efficient to compute than Sigmoid like functions since Relu just needs to pick max(0,$x$) and not perform expensive exponential operations as in Sigmoids
  • Relu : In practice, networks with Relu tend to show better convergence performance than sigmoid. (Krizhevsky et al.)

In other words, Disadvantage:

  • Sigmoid: tend to vanish gradient (cause there is a mechanism to reduce the gradient as "$a$" increase, where "$a$" is the input of a sigmoid function. Gradient of Sigmoid: $S'(a)= S(a)(1-S(a))$. When "$a$" grows to infinite large , $S'(a)= S(a)(1-S(a)) = 1\times(1-1)=0$).

  • Relu : tend to blow up activation (there is no mechanism to constrain the output of the neuron, as "$a$" itself is the output)

  • Relu : Dying Relu problem - if too many activations get below zero then most of the units(neurons) in network with Relu will simply output zero, in other words, die and thereby prohibiting learning.(This can be handled, to some extent, by using Leaky-Relu instead.)

Advantage:

  • Sigmoid: not blowing up activation
  • Relu : not vanishing gradient

In other words, Disadvantage:

  • Sigmoid: tend to vanish gradient (cause there is a mechanism to reduce the gradient as "$a$" increase, where "$a$" is the input of a sigmoid function. Gradient of Sigmoid: $S'(a)= S(a)(1-S(a))$. When "$a$" grows to infinite large , $S'(a)= S(a)(1-S(a)) = 1\times(1-1)=0$).

  • Relu : tend to blow up activation (there is no mechanism to constrain the output of the neuron, as "$a$" itself is the output)

Advantage:

  • Sigmoid: not blowing up activation
  • Relu : not vanishing gradient
  • Relu : More computationally efficient to compute than Sigmoid like functions since Relu just needs to pick max(0,$x$) and not perform expensive exponential operations as in Sigmoids
  • Relu : In practice, networks with Relu tend to show better convergence performance than sigmoid. (Krizhevsky et al.)

Disadvantage:

  • Sigmoid: tend to vanish gradient (cause there is a mechanism to reduce the gradient as "$a$" increase, where "$a$" is the input of a sigmoid function. Gradient of Sigmoid: $S'(a)= S(a)(1-S(a))$. When "$a$" grows to infinite large , $S'(a)= S(a)(1-S(a)) = 1\times(1-1)=0$).

  • Relu : tend to blow up activation (there is no mechanism to constrain the output of the neuron, as "$a$" itself is the output)

  • Relu : Dying Relu problem - if too many activations get below zero then most of the units(neurons) in network with Relu will simply output zero, in other words, die and thereby prohibiting learning.(This can be handled, to some extent, by using Leaky-Relu instead.)

added 14 characters in body
Source Link

Advantage:

  • Sigmoid: not blowing up activation
  • Relu : not vanishing gradient

In other words, Disadvantage:

  • Sigmoid: tend to blow up activation (there is no mechanism to constrain the output of the neuron, as "a" itself is the output)

    Sigmoid: tend to vanish gradient (cause there is a mechanism to reduce the gradient as "$a$" increase, where "$a$" is the input of a sigmoid function. Gradient of Sigmoid: $S'(a)= S(a)(1-S(a))$. When "$a$" grows to infinite large , $S'(a)= S(a)(1-S(a)) = 1\times(1-1)=0$).

  • Relu : tend to vanish gradient (cause there is a mechanism to reduce the gradient as "a" increase, where "a" is the input of a sigmoid function. Gradient of Sigmoid: $S'(a)= S(a)(1-S(a))$. When "a" grows to infinite large , $S'(a)= S(a)(1-S(a)) = 1(1-1)=0$).

    Relu : tend to blow up activation (there is no mechanism to constrain the output of the neuron, as "$a$" itself is the output)

Advantage:

  • Sigmoid: not blowing up activation
  • Relu : not vanishing gradient

In other words, Disadvantage:

  • Sigmoid: tend to blow up activation (there is no mechanism to constrain the output of the neuron, as "a" itself is the output)
  • Relu : tend to vanish gradient (cause there is a mechanism to reduce the gradient as "a" increase, where "a" is the input of a sigmoid function. Gradient of Sigmoid: $S'(a)= S(a)(1-S(a))$. When "a" grows to infinite large , $S'(a)= S(a)(1-S(a)) = 1(1-1)=0$).

Advantage:

  • Sigmoid: not blowing up activation
  • Relu : not vanishing gradient

In other words, Disadvantage:

  • Sigmoid: tend to vanish gradient (cause there is a mechanism to reduce the gradient as "$a$" increase, where "$a$" is the input of a sigmoid function. Gradient of Sigmoid: $S'(a)= S(a)(1-S(a))$. When "$a$" grows to infinite large , $S'(a)= S(a)(1-S(a)) = 1\times(1-1)=0$).

  • Relu : tend to blow up activation (there is no mechanism to constrain the output of the neuron, as "$a$" itself is the output)

add latex
Source Link
Silverfish
  • 23.9k
  • 28
  • 105
  • 206

Advantage:

  • Sigmoid: not blowing up activation
  • Relu : not vanishing gradient

In other words, Disadvantage:

  • Sigmoid: tend to blow up activation (there is no mechanism to constrain the output of the neuron, as "a" itself is the output)
  • Relu : tend to vanish gradient (cause there is a mechanism to reduce the gradient as "a" increase, where "a" is the input of a sigmoid function. Gradient of Sigmoid: S'(a)= S(a)(1-S(a))$S'(a)= S(a)(1-S(a))$. When "a" grows to infinite large , S'(a)= S(a)(1-S(a)) = 1*(1-1)=0 $S'(a)= S(a)(1-S(a)) = 1(1-1)=0$).

Advantage:

  • Sigmoid: not blowing up activation
  • Relu : not vanishing gradient

In other words, Disadvantage:

  • Sigmoid: tend to blow up activation (there is no mechanism to constrain the output of the neuron, as "a" itself is the output)
  • Relu : tend to vanish gradient (cause there is a mechanism to reduce the gradient as "a" increase, where "a" is the input of a sigmoid function. Gradient of Sigmoid: S'(a)= S(a)(1-S(a)). When "a" grows to infinite large , S'(a)= S(a)(1-S(a)) = 1*(1-1)=0 )

Advantage:

  • Sigmoid: not blowing up activation
  • Relu : not vanishing gradient

In other words, Disadvantage:

  • Sigmoid: tend to blow up activation (there is no mechanism to constrain the output of the neuron, as "a" itself is the output)
  • Relu : tend to vanish gradient (cause there is a mechanism to reduce the gradient as "a" increase, where "a" is the input of a sigmoid function. Gradient of Sigmoid: $S'(a)= S(a)(1-S(a))$. When "a" grows to infinite large , $S'(a)= S(a)(1-S(a)) = 1(1-1)=0$).
added 66 characters in body
Source Link
Loading
Source Link
Loading