ML: RELU versus ln(1+exp(X)) I wasn't able to find this question in SE, possibly as I am new to ML:
When deciding on an activation function, why is RELU preferred over a function such as ln(1 + exp(x))?  The former has zero sensitivity for negative values and so one might get a "dead" neuron. With ln(1 + exp(x)) at least one wouldn't step into a fully dead area.
Does it have to do with being able to create simpler analytic derivatives for backprop or something?
 A: There are undoubtedly many way to provide such a justification. The simplest is that a logic gate can be constructed from rectifiers (diodes), but not from sigmoidal functions. Read a bit about logic gates here. From that "coincidence" or if you wish "rediscovery of the wheel," when neurons are joined so that they can duplicate "logic gates" they are also joined in an "efficient" and "robust" fashion. A sigmoidal function is like a rectifier with a back current leak, it will not form a clean logic gate, but rather, a dirty inefficient one. Granted, I have used electronic jargon here. The neural net equivalent language may not be even that accessible.
A: As you mentioned, RELU has the dying problem which we cannot ignore. However, some improved versions like Leaky RELU, Parametric RELU have been invented, and it's no longer been as a considerable disadvantage of RELU.
For more references, you can look at these sites:

*

*Dying ReLU: Causes and Solutions (Leaky ReLU)


*A Practical Guide to ReLU
