7
$\begingroup$

selu

See above this new activation function for MLPs (the paper just came out yesterday). The equation is found in the paper on page 3. However cannot find the backprop. formula (chain-rule result from backprop). Does anyone know?

Selu-Paper

$\endgroup$
4
  • $\begingroup$ What is a "backpropagation formula"? Are you talking about the derivative of the function? $\endgroup$
    – Coolness
    Jun 10, 2017 at 15:22
  • $\begingroup$ This is CrossValidated, perhaps you were looking for WolframAlpha? $\endgroup$
    – GeoMatt22
    Jun 10, 2017 at 17:38
  • $\begingroup$ @Coolness, yes I am looking for the chain rule result (backprop). $\endgroup$ Jun 10, 2017 at 18:15
  • $\begingroup$ This is basic calculus. It ought to be moved to MSE. $\endgroup$
    – nth
    Jun 10, 2017 at 18:59

2 Answers 2

9
$\begingroup$

Ok, let's try it myself. For the backward pass we get:

$$ \frac{\partial E}{\partial Y}\frac{\partial Y}{\partial X} = λ, x>0 $$

$$ \frac{\partial E}{\partial Y}\frac{\partial Y}{\partial X} = λ * a * e ^x , x=<0 $$

with $ λ=1.0507, a=1.6733$

$\endgroup$
3
  • $\begingroup$ Forgot the $\lambda$. $\endgroup$
    – GeoMatt22
    Jun 10, 2017 at 19:27
  • $\begingroup$ yeah, that is why it was worth asking this question :) $\endgroup$ Jun 11, 2017 at 8:11
  • $\begingroup$ Hi, not totally familiar with math notation (six years since graduating Uni...) Does this mean to only multiply e^x by alpha and lambda only if x <= 0? (Edit: fixed missing =) $\endgroup$
    – Zeroth
    Aug 6, 2017 at 17:56
0
$\begingroup$

The derivative (d) of the Selu function can be found from both the input (x) into and the output (y) from the Selu function.

To find derivative from the input:

$$ d = seluDerivative(x) = \begin{cases} \lambda & \text{if } x > 0\\ \lambda\alpha e^x & \text{if } x \leqslant 0\\ \end{cases} $$

The problem with using this function in a backprop is that you might not want to save the temporary x value made from the forwardprop and recalculating it would be slow.

Luckily the Selu function is fully invertible so we can use this to find the derivative with y from the Selu function:

$$ y = selu(x) = \lambda \begin{cases} x & \text{if } x > 0\\ \alpha e^x - \alpha & \text{if } x \leqslant 0\\ \end{cases} $$

$$ x = seluInverse(y) = \begin{cases} y \over \lambda & \text{if } y > 0\\ \ln \left( y + \lambda\alpha \over \lambda\alpha \right) & \text{if } y \leqslant 0\\ \end{cases} $$

$$ d = seluInverseDerivative(y) = \begin{cases} \lambda & \text{if } y > 0\\ y + \lambda\alpha & \text{if } y \leqslant 0\\ \end{cases} $$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.