6
$\begingroup$

I am reading Stanford's tutorial on the subject, and I have reached this part, "Training a Neural Network". So far so good. I understand pretty much everything.

I would like to change the ReLU he is using there, with a Leaky ReLU. My question, is, do I have to change the way he is doing the back-propagation? How do these derivatives going to change if I use a Leaky ReLU?

Any paper that's states exactly how back prop is done when we have a Leaky ReLU?

$\endgroup$
7
$\begingroup$

The derivative of a ReLU is zero for x < 0 and one for x > 0. If the leaky ReLU has slope, say 0.5, for negative values, the derivative will be 0.5 for x < 0 and 1 for x > 0.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.