13
$\begingroup$

I am reading Stanford's tutorial on the subject, and I have reached this part, "Training a Neural Network". So far so good. I understand pretty much everything.

I would like to change the ReLU he is using there, with a Leaky ReLU. My question, is, do I have to change the way he is doing the back-propagation? How do these derivatives going to change if I use a Leaky ReLU?

Any paper that states exactly how back prop is done when we have a Leaky ReLU?

$\endgroup$

3 Answers 3

15
$\begingroup$

The derivative of a ReLU is zero for x < 0 and one for x > 0. If the leaky ReLU has slope, say 0.5, for negative values, the derivative will be 0.5 for x < 0 and 1 for x > 0.

$\endgroup$
3
$\begingroup$

For some $c$, we have the leaky relu $f(x)$ $$\begin{align} f(x)&=\begin{cases} x & x \ge 0\\ cx & x<0 \end{cases}\\ f^\prime(x)&=\begin{cases} 1 & x > 0 \\ c &x<0 \end{cases} \end{align} .$$ The leaky ReLU function is not differentiable at $x=0$ unless $c=1$.

Usually, one chooses $0<c<1$. The special case of $c=0$ is an ordinary ReLU, and the special case of $c=1$ is just the identity function. Choosing $c>1$ implies that the composition of many such layers might exhibit exploding gradients, which is undesirable. Also, choosing $c<0$ makes $f$ a non-monotonic function shaped something like a $V$. Non-monotonic functions have recently become more popular (e.g. mish and swish), but I'm not aware of a study of a non-monotonic leaky ReLU.

$\endgroup$
1
$\begingroup$

If $\alpha$ is the slope for negative $x$, a compact way to write the derivative is

$$\alpha +(1-\alpha)H(z)$$

where $H(z)$ is the Heaviside step function. The non-leaky ReLU is case $\alpha=0$ with derivative $H(z)$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.