19
$\begingroup$

The ReLU function is commonly used as an activation function in machine learning, as well, as its modifications (ELU, leaky ReLU).

The overall idea of these functions is the same: before x = 0 the value of the function is small (its limit to infinity is zero or -1), after x = 0 the function grows proportionally to x.

The exponent function (e^x or e^x-1) has similar behavior, and its derivative in x = 0 is greater than for sigmoid.

The visualization below illustrates the exponent in comparison with ReLU and sigmoid activation functions.

The comparison of the exponent with some popular activations

So, why the simple function y=e^x is not used as an activation function in neural networks?

Improve this question
$\endgroup$
3
  • 3
    $\begingroup$ You can. There is nothing stopping you. If you are interested in a smooth variant of the ReLU function, why not use the Softplus function? $\endgroup$
    – mhdadk
    Commented Jun 26, 2021 at 16:33
  • 3
    $\begingroup$ @mhdadk, I am not asking, if I can do this or not. I'm asking about the drawback(s) of the relatively naive solution to the problem of activation functions. $\endgroup$
    – MefAldemisov
    Commented Jun 26, 2021 at 16:38
  • 4
    $\begingroup$ If you view Neural Networks in the abstract as a sort of black-box tool for function approximation, a good toy problem in this domain is to consider the convergence of continued exponentials. Bender and Vinson wrote about this in the context of pertubation theory, where you use only a small number of terms of the Taylor expansion. The upshot was that the convergence, while quite fascinating, has an intricate structure and doesn't work everywhere, which explains a bit why this kind of thing might not be suitable as an activation function. $\endgroup$
    – Cade Reinberger
    Commented Jun 28, 2021 at 3:29

1 Answer 1

Reset to default
33
$\begingroup$

I think the most prominent reason is stability. Think about having consequent layers with exponential activation, and what happens to the output when you input a small number to the NN (e.g. $x=1$), the forward calculation will look like: $$o=\exp(\exp(\exp(\exp(1))))\approx e^{3814279}$$

It can go crazy very quickly and I don't think you can train deep networks with this activation function unless you add other mechanisms like clipping.

Improve this answer
$\endgroup$
20
  • 7
    $\begingroup$ Well, when using floating point this quickly becomes an “indicator function” $f(x) = \begin{cases}0 & \text{if $x\leq 0$} \\ \infty &\text{otherwise.}\end{cases}$ That kind of “function” can actually be used with nonsmooth convex-optimisation algorithms like Douglas-Rachford (which don't really operate on the function itself or its gradient, but instead on proximals and convex conjugates). For GD-based nonlinear optimisation as used in deep learning, this is not an option. $\endgroup$
    – leftaroundabout
    Commented Jun 27, 2021 at 13:34
  • 4
    $\begingroup$ Even functions that saturate at 1 (sigmoids, arctan) have similar issues; maybe they are simply less apparent because they don't result in an inf, NaN, or error message. $\endgroup$
    – Federico Poloni
    Commented Jun 27, 2021 at 14:17
  • $\begingroup$ that's a noteworthy point, but if this were the only problem, then defining the activation as g(x)= e^(x-1) would solve it since g(g(g(1))) is just 1. $\endgroup$
    – KishKash
    Commented Jul 4, 2021 at 15:45
  • 1
    $\begingroup$ The input $1$ is just an example, it could be $2$ as well. The point is with a very small input, the network can output very large numbers in a handful of layers. $\endgroup$
    – gunes
    Commented Jul 4, 2021 at 15:47
  • 1
    $\begingroup$ So, the formula simplifies what will happen in the neural network. If you have values in [-1, 1] in the first hidden layer's output, the next layer's inputs are going to be around $[1/e, e]$. Then, you have a linear transformation. You don't know what is going to happen in the learning, so, you need to assume things can go bad. In it's simplest form, the values will be around the range $[w/e, we]$ (ignore the bias for simplicity). $w$ is determined in the learning, it could be small or large. For simplicity, if you assume $w$ is around $1$, the next layer's outputs are in $[e^{1/e}, e^e]$ ... $\endgroup$
    – gunes
    Commented Apr 11 at 20:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.