Approximating leaky ReLU with a differentiable function The ReLU activation function of a neural net can be approximated by the softplus function, which is differentiable. How would you approximate the leaky ReLU with a differentiable function?
 A: The softplus function is commonly described as a smooth approximation of the standard ReLU:
$$s(x) = \log(1 + e^x)$$

The leaky ReLU (with leak coefficient $\alpha$) is:
$$r_L(x) = \max \{ \alpha x, x\}$$
We can also write this as:
$$r_L(x) = \alpha x + (1-\alpha) \max\{0, x\}$$
Note that $\max\{0, x\}$ is the standard ReLU. So, we can construct a smooth approximation to the leaky ReLU by substituting in the softplus function:
$$s_L(x) = \alpha x + (1-\alpha) \log(1 + e^x)$$

A: I stumbled on this on accident, not sure if this would be useful but try this weird function I thought of:
(1/20)x(e^arctan(x))
Edit: let's put a constant of say (1/20) in front to prevent the slope beign greater than one
arctan goes to plus or minus pi/2 at values |x|>3 so we have on the left hand side xe^(-pi/2)= x(small constant) and on the right side xe^(pi/2) = x(large constant) which resembles the leaky-relu activation function
plotted on wolfram alpha:
https://www.wolframalpha.com/input/?i=x*%28e%5Earctan%28x%29%29
A: Leaky RELU is defined as:

LogSumExp is a smooth approximation of the max function. So I would go with:

