It is often mentioned that rectified linear units (ReLU) have superseded softplus units because they are linear and faster to compute.

Does softplus it still have the advantage of inducing sparsity or is that restricted to the ReLU?

The reason I ask is it I wonder about negative consequences of the zero slope of the ReLU. Doesn't this property "trap" units at zero where it might be beneficial to give them the possibility of reactivation?

  • $\begingroup$ did you ever found out the answer to this? $\endgroup$ – Charlie Parker Oct 11 '16 at 0:15

I found an answer to your question in the Section 6.3.3 of the Deep Learning book. (Goodfellow et. al, 2016):

The use of softplus is generally discouraged. ... one might expect it to have advantage over the rectifier due to being differentiable everywhere or due to saturating less completely, but empirically it does not.

As a reference to support this claim they cite the paper Deep Sparse Rectifier Neural Networks (Glorot et. al, 2011).


ReLUs can indeed be permanently switched off, particularly under high learning rates. This is a motivation behind leaky ReLU, and ELU activations, both of which have non-zero gradient almost everywhere.

Leaky ReLU is a piecewise linear function, just as for ReLU, so quick to compute. ELU has the advantage over softmax and ReLU that it's mean output is closer to zero, which improves learning.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.