0
$\begingroup$

I came across SELU, and I was wondering how to derive $alpha$ and $lambda$ in the function, because I wasn't able to understand the math in the reseach article. I did, however, find this website that drew the following values from the paper: $$alpha = 1.673263...$$ $$lambda = 1.050701...$$ But, now I'm slightly confused because I thought self-normalizing meant the function would adjust its $alpha$ and $lambda$ values over time with the influx of training data. Or have I misread something? If I could provide a link to a video or a website or if I could manage an explanation of the functions which define the alpha and lambda values I'd really appreciate it.

$\endgroup$
0
$\begingroup$

I don't claim to understand the full proof provided in the paper. However, I do know that $\alpha$ and $\lambda$ are constants, not learned.

I thought self-normalizing meant the function would adjust its alpha and lambda values over time with the influx of training data.

No, the term self-normalizing refers to the fact that using this activation causes the distribution of activations to roughly adhere to the standard normal distribution.

$\endgroup$

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.