# What is the derivative of Leaky ReLU?

I am reading Stanford's tutorial on the subject, and I have reached this part, "Training a Neural Network". So far so good. I understand pretty much everything.

I would like to change the ReLU he is using there, with a Leaky ReLU. My question, is, do I have to change the way he is doing the back-propagation? How do these derivatives going to change if I use a Leaky ReLU?

Any paper that states exactly how back prop is done when we have a Leaky ReLU?

For some $$c$$, we have the leaky relu $$f(x)$$ \begin{align} f(x)&=\begin{cases} x & x \ge 0\\ cx & x<0 \end{cases}\\ f^\prime(x)&=\begin{cases} 1 & x > 0 \\ c &x<0 \end{cases} \end{align} . The leaky ReLU function is not differentiable at $$x=0$$ unless $$c=1$$.
Usually, one chooses $$0. The special case of $$c=0$$ is an ordinary ReLU, and the special case of $$c=1$$ is just the identity function. Choosing $$c>1$$ implies that the composition of many such layers might exhibit exploding gradients, which is undesirable. Also, choosing $$c<0$$ makes $$f$$ a non-monotonic function shaped something like a $$V$$. Non-monotonic functions have recently become more popular (e.g. mish and swish), but I'm not aware of a study of a non-monotonic leaky ReLU.
If $$\alpha$$ is the slope for negative $$x$$, a compact way to write the derivative is
$$\alpha +(1-\alpha)H(z)$$
where $$H(z)$$ is the Heaviside step function. The non-leaky ReLU is case $$\alpha=0$$ with derivative $$H(z)$$.