I am reading Stanford's tutorial on the subject, and I have reached this part, "Training a Neural Network". So far so good. I understand pretty much everything.
I would like to change the ReLU he is using there, with a Leaky ReLU. My question, is, do I have to change the way he is doing the back-propagation? How do these derivatives going to change if I use a Leaky ReLU?
Any paper that states exactly how back prop is done when we have a Leaky ReLU?
The derivative of a ReLU is zero for x < 0 and one for x > 0. If the leaky ReLU has slope, say 0.5, for negative values, the derivative will be 0.5 for x < 0 and 1 for x > 0.
For some $c$, we have the leaky relu $f(x)$ $$\begin{align} f(x)&=\begin{cases} x & x \ge 0\\ cx & x<0 \end{cases}\\ f^\prime(x)&=\begin{cases} 1 & x > 0 \\ c &x<0 \end{cases} \end{align} .$$ The leaky ReLU function is not differentiable at $x=0$ unless $c=1$.
Usually, one chooses $0<c<1$. The special case of $c=0$ is an ordinary ReLU, and the special case of $c=1$ is just the identity function. Choosing $c>1$ implies that the composition of many such layers might exhibit exploding gradients, which is undesirable. Also, choosing $c<0$ makes $f$ a non-monotonic function shaped something like a $V$. Non-monotonic functions have recently become more popular (e.g. mish and swish), but I'm not aware of a study of a non-monotonic leaky ReLU.
If $\alpha$ is the slope for negative $x$, a compact way to write the derivative is
$$\alpha +(1-\alpha)H(z)$$
where $H(z)$ is the Heaviside step function. The non-leaky ReLU is case $\alpha=0$ with derivative $H(z)$.
