What are the benefits of using ReLU over softplus as activation functions? It is often mentioned that rectified linear units (ReLU) have superseded softplus units because they are linear and faster to compute.
Does softplus it still have the advantage of inducing sparsity or is that restricted to the ReLU?
The reason I ask is it I wonder about negative consequences of the zero slope of the ReLU. Doesn't this property "trap" units at zero where it might be beneficial to give them the possibility of reactivation?
 A: ReLUs can indeed be permanently switched off, particularly under high learning rates. This is a motivation behind leaky ReLU, and ELU activations, both of which have non-zero gradient almost everywhere.
Leaky ReLU is a piecewise linear function, just as for ReLU, so quick to compute. ELU has the advantage over softplus and ReLU that its mean output is closer to zero, which improves learning.
A: The main reason ReLU works better than Softplus is that for ReLU we have the idea of sparsity in the model. This means that some of the neurons of the model output zero which does not have any effect for the next layers. this idea is something like Dropout. Neurons in hidden layers learn hidden concepts. If the input does not contain the corresponding concept, some neurons will output zero and they will not be engaged in the calculations of the next layers. This idea cannot be in Softplus, because the output cannot be zero like ReLU.
A: I found an answer to your question in the Section 6.3.3 of the Deep Learning book. (Goodfellow et. al, 2016):

The use of softplus is generally discouraged. ... one might expect it to have advantage over the rectifier due to being differentiable everywhere or due to saturating less completely, but empirically it does not.

As a reference to support this claim they cite the paper Deep Sparse Rectifier Neural Networks (Glorot et. al, 2011).
