What is the backpropagation formula for Selu activation function? 
See above this new activation function for MLPs (the paper just came out yesterday). The equation is found in the paper on page 3. However cannot find the backprop. formula (chain-rule result from backprop). Does anyone know?
Selu-Paper
 A: Ok, let's try it myself. For the backward pass we get:
$$
\frac{\partial E}{\partial Y}\frac{\partial Y}{\partial X} = λ, x>0
$$
$$
\frac{\partial E}{\partial Y}\frac{\partial Y}{\partial X} = λ * a * e ^x , x=<0
$$
with $ λ=1.0507, a=1.6733$
A: The derivative (d) of the Selu function can be found from both the input (x) into and the output (y) from the Selu function.
To find derivative from the input:
$$
d = seluDerivative(x) =
\begin{cases}
\lambda & \text{if } x > 0\\
\lambda\alpha e^x & \text{if } x \leqslant 0\\
\end{cases}
$$
The problem with using this function in a backprop is that you might not want to save the temporary x value made from the forwardprop and recalculating it would be slow.
Luckily the Selu function is fully invertible so we can use this to find the derivative with y from the Selu function:
$$
y = selu(x) = \lambda
\begin{cases}
x & \text{if } x > 0\\
\alpha e^x - \alpha & \text{if } x \leqslant 0\\
\end{cases}
$$
$$
x = seluInverse(y) =
\begin{cases}
y \over \lambda & \text{if } y > 0\\
\ln \left(  y + \lambda\alpha \over \lambda\alpha \right) & \text{if } y \leqslant 0\\
\end{cases}
$$
$$
d = seluInverseDerivative(y) =
\begin{cases}
\lambda & \text{if } y > 0\\
y + \lambda\alpha & \text{if } y \leqslant 0\\
\end{cases}
$$
