What is the derivative of Leaky ReLU? I am reading Stanford's tutorial on the subject, and I have reached this part, "Training a Neural Network". So far so good. I understand pretty much everything. 
I would like to change the ReLU he is using there, with a Leaky ReLU. My question, is, do I have to change the way he is doing the back-propagation? How do these derivatives going to change if I use a Leaky ReLU?
Any paper that states exactly how back prop is done when we have a Leaky ReLU? 
 A: For some $c$, we have the leaky relu $f(x)$
$$\begin{align}
f(x)&=\begin{cases}
x & x \ge 0\\
cx & x<0
\end{cases}\\
f^\prime(x)&=\begin{cases}
1 & x > 0 \\
c &x<0
\end{cases}
\end{align}
.$$
The leaky ReLU function is not differentiable at $x=0$ unless $c=1$.
Usually, one chooses $0<c<1$. The special case of $c=0$ is an ordinary ReLU, and the special case of $c=1$ is just the identity function. Choosing $c>1$ implies that the composition of many such layers might exhibit exploding gradients, which is undesirable. Also, choosing $c<0$ makes $f$ a non-monotonic function shaped something like a $V$. Non-monotonic functions have recently become more popular (e.g. mish and swish), but I'm not aware of a study of a non-monotonic leaky ReLU.
A: The derivative of a ReLU is zero for x < 0 and one for x > 0. If the leaky ReLU has slope, say 0.5, for negative values, the derivative will be 0.5 for x < 0 and 1 for x > 0.
A: If $\alpha$ is the slope for negative $x$, a compact way to write the derivative is
$$\alpha +(1-\alpha)H(z)$$
where $H(z)$ is the Heaviside step function. The non-leaky ReLU is case $\alpha=0$ with derivative $H(z)$.
