# What is the backpropagation formula for Selu activation function?

See above this new activation function for MLPs (the paper just came out yesterday). The equation is found in the paper on page 3. However cannot find the backprop. formula (chain-rule result from backprop). Does anyone know?

Selu-Paper

• What is a "backpropagation formula"? Are you talking about the derivative of the function? Jun 10, 2017 at 15:22
• This is CrossValidated, perhaps you were looking for WolframAlpha? Jun 10, 2017 at 17:38
• @Coolness, yes I am looking for the chain rule result (backprop). Jun 10, 2017 at 18:15
• This is basic calculus. It ought to be moved to MSE.
– nth
Jun 10, 2017 at 18:59

Ok, let's try it myself. For the backward pass we get:

$$\frac{\partial E}{\partial Y}\frac{\partial Y}{\partial X} = λ, x>0$$

$$\frac{\partial E}{\partial Y}\frac{\partial Y}{\partial X} = λ * a * e ^x , x=<0$$

with $λ=1.0507, a=1.6733$

• Forgot the $\lambda$. Jun 10, 2017 at 19:27
• yeah, that is why it was worth asking this question :) Jun 11, 2017 at 8:11
• Hi, not totally familiar with math notation (six years since graduating Uni...) Does this mean to only multiply e^x by alpha and lambda only if x <= 0? (Edit: fixed missing =) Aug 6, 2017 at 17:56

The derivative (d) of the Selu function can be found from both the input (x) into and the output (y) from the Selu function.

To find derivative from the input:

$$d = seluDerivative(x) = \begin{cases} \lambda & \text{if } x > 0\\ \lambda\alpha e^x & \text{if } x \leqslant 0\\ \end{cases}$$

The problem with using this function in a backprop is that you might not want to save the temporary x value made from the forwardprop and recalculating it would be slow.

Luckily the Selu function is fully invertible so we can use this to find the derivative with y from the Selu function:

$$y = selu(x) = \lambda \begin{cases} x & \text{if } x > 0\\ \alpha e^x - \alpha & \text{if } x \leqslant 0\\ \end{cases}$$

$$x = seluInverse(y) = \begin{cases} y \over \lambda & \text{if } y > 0\\ \ln \left( y + \lambda\alpha \over \lambda\alpha \right) & \text{if } y \leqslant 0\\ \end{cases}$$

$$d = seluInverseDerivative(y) = \begin{cases} \lambda & \text{if } y > 0\\ y + \lambda\alpha & \text{if } y \leqslant 0\\ \end{cases}$$