# Why pure exponent is not used as activation function for neural networks?

The ReLU function is commonly used as an activation function in machine learning, as well, as its modifications (ELU, leaky ReLU).

The overall idea of these functions is the same: before x = 0 the value of the function is small (its limit to infinity is zero or -1), after x = 0 the function grows proportionally to x.

The exponent function (e^x or e^x-1) has similar behavior, and its derivative in x = 0 is greater than for sigmoid.

The visualization below illustrates the exponent in comparison with ReLU and sigmoid activation functions.

So, why the simple function y=e^x is not used as an activation function in neural networks?

• You can. There is nothing stopping you. If you are interested in a smooth variant of the ReLU function, why not use the Softplus function? Commented Jun 26, 2021 at 16:33
• @mhdadk, I am not asking, if I can do this or not. I'm asking about the drawback(s) of the relatively naive solution to the problem of activation functions. Commented Jun 26, 2021 at 16:38
• If you view Neural Networks in the abstract as a sort of black-box tool for function approximation, a good toy problem in this domain is to consider the convergence of continued exponentials. Bender and Vinson wrote about this in the context of pertubation theory, where you use only a small number of terms of the Taylor expansion. The upshot was that the convergence, while quite fascinating, has an intricate structure and doesn't work everywhere, which explains a bit why this kind of thing might not be suitable as an activation function. Commented Jun 28, 2021 at 3:29

I think the most prominent reason is stability. Think about having consequent layers with exponential activation, and what happens to the output when you input a small number to the NN (e.g. $$x=1$$), the forward calculation will look like: $$o=\exp(\exp(\exp(\exp(1))))\approx e^{3814279}$$
• Well, when using floating point this quickly becomes an “indicator function” $f(x) = \begin{cases}0 & \text{if$x\leq 0$} \\ \infty &\text{otherwise.}\end{cases}$ That kind of “function” can actually be used with nonsmooth convex-optimisation algorithms like Douglas-Rachford (which don't really operate on the function itself or its gradient, but instead on proximals and convex conjugates). For GD-based nonlinear optimisation as used in deep learning, this is not an option. Commented Jun 27, 2021 at 13:34
• Even functions that saturate at 1 (sigmoids, arctan) have similar issues; maybe they are simply less apparent because they don't result in an inf, NaN, or error message. Commented Jun 27, 2021 at 14:17
• The input $1$ is just an example, it could be $2$ as well. The point is with a very small input, the network can output very large numbers in a handful of layers. Commented Jul 4, 2021 at 15:47
• So, the formula simplifies what will happen in the neural network. If you have values in [-1, 1] in the first hidden layer's output, the next layer's inputs are going to be around $[1/e, e]$. Then, you have a linear transformation. You don't know what is going to happen in the learning, so, you need to assume things can go bad. In it's simplest form, the values will be around the range $[w/e, we]$ (ignore the bias for simplicity). $w$ is determined in the learning, it could be small or large. For simplicity, if you assume $w$ is around $1$, the next layer's outputs are in $[e^{1/e}, e^e]$ ... Commented Apr 11 at 20:23