The ReLU function is commonly used as an activation function in machine learning, as well, as its modifications (ELU, leaky ReLU).
The overall idea of these functions is the same: before
x = 0 the value of the function is small (its limit to infinity is zero or
x = 0 the function grows proportionally to x.
The exponent function (
e^x-1) has similar behavior, and its derivative in
x = 0 is greater than for sigmoid.
The visualization below illustrates the exponent in comparison with ReLU and sigmoid activation functions.
So, why the simple function
y=e^x is not used as an activation function in neural networks?