The ReLU function is commonly used as an activation function in machine learning, as well, as its modifications (ELU, leaky ReLU).

The overall idea of these functions is the same: before x = 0 the value of the function is small (its limit to infinity is zero or -1), after x = 0 the function grows proportionally to x.

The exponent function (e^x or e^x-1) has similar behavior, and its derivative in x = 0 is greater than for sigmoid.

The visualization below illustrates the exponent in comparison with ReLU and sigmoid activation functions.

The comparison of the exponent with some popular activations

So, why the simple function y=e^x is not used as an activation function in neural networks?

  • 2
    $\begingroup$ You can. There is nothing stopping you. If you are interested in a smooth variant of the ReLU function, why not use the Softplus function? $\endgroup$
    – mhdadk
    Jun 26 at 16:33
  • 2
    $\begingroup$ @mhdadk, I am not asking, if I can do this or not. I'm asking about the drawback(s) of the relatively naive solution to the problem of activation functions. $\endgroup$ Jun 26 at 16:38
  • 2
    $\begingroup$ If you view Neural Networks in the abstract as a sort of black-box tool for function approximation, a good toy problem in this domain is to consider the convergence of continued exponentials. Bender and Vinson wrote about this in the context of pertubation theory, where you use only a small number of terms of the Taylor expansion. The upshot was that the convergence, while quite fascinating, has an intricate structure and doesn't work everywhere, which explains a bit why this kind of thing might not be suitable as an activation function. $\endgroup$ Jun 28 at 3:29

I think the most prominent reason is stability. Think about having consequent layers with exponential activation, and what happens to the output when you input a small number to the NN (e.g. $x=1$), the forward calculation will look like: $$o=\exp(\exp(\exp(\exp(1))))\approx e^{3814279}$$

It can go crazy very quickly and I don't think you can train deep networks with this activation function unless you add other mechanisms like clipping.

  • 5
    $\begingroup$ Well, when using floating point this quickly becomes an “indicator function” $f(x) = \begin{cases}0 & \text{if $x\leq 0$} \\ \infty &\text{otherwise.}\end{cases}$ That kind of “function” can actually be used with nonsmooth convex-optimisation algorithms like Douglas-Rachford (which don't really operate on the function itself or its gradient, but instead on proximals and convex conjugates). For GD-based nonlinear optimisation as used in deep learning, this is not an option. $\endgroup$ Jun 27 at 13:34
  • 4
    $\begingroup$ Even functions that saturate at 1 (sigmoids, arctan) have similar issues; maybe they are simply less apparent because they don't result in an inf, NaN, or error message. $\endgroup$ Jun 27 at 14:17
  • $\begingroup$ that's a noteworthy point, but if this were the only problem, then defining the activation as g(x)= e^(x-1) would solve it since g(g(g(1))) is just 1. $\endgroup$
    – KishKash
    Jul 4 at 15:45
  • 1
    $\begingroup$ The input $1$ is just an example, it could be $2$ as well. The point is with a very small input, the network can output very large numbers in a handful of layers. $\endgroup$
    – gunes
    Jul 4 at 15:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.