0
$\begingroup$

I came across SELU, and I was wondering how to derive $alpha$ and $lambda$ in the function, because I wasn't able to understand the math in the reseach article. I did, however, find this website that drew the following values from the paper: $$alpha = 1.673263...$$ $$lambda = 1.050701...$$ But, now I'm slightly confused because I thought self-normalizing meant the function would adjust its $alpha$ and $lambda$ values over time with the influx of training data. Or have I misread something? If I could provide a link to a video or a website or if I could manage an explanation of the functions which define the alpha and lambda values I'd really appreciate it.

$\endgroup$
0
$\begingroup$

I don't claim to understand the full proof provided in the paper. However, I do know that $\alpha$ and $\lambda$ are constants, not learned.

I thought self-normalizing meant the function would adjust its alpha and lambda values over time with the influx of training data.

No, the term self-normalizing refers to the fact that using this activation causes the distribution of activations to roughly adhere to the standard normal distribution.

$\endgroup$

Your Answer

By clicking "Post Your Answer", you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.