# Backpropogation Derivatives

I've been working on trying to understand the backpropogation algorithm and the calculus behind it, and in my work I have stumbled across a sort of odd situation. I am just practicing on a 1 input, 1 output, 1 hidden node network, and after taking the derivatives of the second weight in the network in the connection from the hidden to the out put node, I got this...

I am using a simple quadratic cost function of $$C(a) = (a^l-y)^2,$$ where $$a$$ is the activation and l is used to indicate which layer relative to the output layer. I am using the sigmoid function for my non-linearity. So from that I have $$a^l(z^l) = \sigma(z^l)$$ and for $$z$$

$$z^l(w^l) = a^{(l-1)}\cdot w^l$$

and so, using the chain rule,

$$\frac{\partial C}{\partial w^l} = \frac{\partial C}{\partial a^l} \cdot \frac{\partial a^l}{\partial z^l} \cdot \frac{\partial z^l}{\partial w^l} = 2(a^l-y)\cdot (\sigma(z^l)\cdot (1-\sigma(z^l)))\cdot a^{l-1}$$

Now I may have done this wrong, but I am fairly confident in my work here. So I have defined the variables above so that

$$a^l = 1,\; y = 0,\; z^l=50, \; C=1, \;w^l=0.8,\;a^{l-1}=62.5.$$

SO clearly there is work to be done on this weight, but if you plug each value into the derivation above you get 0 due to the derivation a with respect to z and the sigmoid function derivation. This makes no sense, there is clearly a negative gradient to be added, but it comes out to be zero why is that?

Thanks is advance, additionally if anyone has some resources that may better help me understand it would be greatly appreciated if you could share them here

** Note the notation is probably not perfect, so feel free to change it.

Actually, $$a^l=\sigma(z^l)=\frac{1}{1+\exp(-z^l)}=\frac{1}{1+\exp(-50)}\neq 1$$

but since $$\exp(-50)$$ is a very small number, it underflows and the result can be $$1$$ in finite precision. So, This is more of a numerical issue than theoretical. You should normalize your inputs to not come across these types of problems.

P.S. it should be $$\sigma(1-\sigma)$$, not $$\sigma-(1-\sigma)$$

I am just practicing on a 1 input, 1 output, 1 hidden node network

That means your network $$f: \mathbb{R} \rightarrow \mathbb{R}$$ looks like this,

\begin{align*} f(x) & = a^{(3)} && \text{Act. in the last/3^{rd}/output layer.} \\ & = \sigma(z^{(3)}) \\ & = \sigma(w^{(3)}a^{(2)} + b^{(3)}) && a^{(2)}\text{ is the act. in the 2^{nd}/hidden layer.} \\ & = \sigma(w^{(3)}\sigma(z^{(2)}) + b^{(3)}) \\ & = \sigma(w^{(3)}\sigma(w^{(2)}a^{(1)} + b^{(2)}) + b^{(3)}) && a^{(1)}\text{ is the act. in the 1^{st}/input layer.} \\ & = \sigma(w^{(3)}\sigma(w^{(2)}x + b^{(2)}) + b^{(3)}) \end{align*} $$\newcommand\d{\partial}$$ Therefore, $$f(x) = \sigma(w^{(3)}\sigma(w^{(2)}x + b^{(2)}) + b^{(3)})$$

Let $$y$$ be the expected $$f(x) = y$$ (but in reality $$f(x) \neq y$$) then the error or (quadratic) cost function is,

$$C(x, f) = (y - f(x))^{2} \text{ and } \nabla C = \left[\frac{\d C}{\d w^{(3)}}, \frac{\d C}{\d b^{(3)}}, \frac{\d C}{\d w^{(2)}}, \frac{\d C}{\d b^{(2)}}\right]$$

Now let's see how we can take these partial derivative,

\begin{align*} \frac{\d C(x, f)}{\d w^{(3)}} & = \frac{\d (y - f(x))^{2}}{\d w^{(3)}} \\ & = \frac{\d (y - f(x))^{2}}{\d w^{(3)}} \\ & = \frac{\d (y - f(x))^{2}}{\d(y - f(x))} \frac{\d(y - f(x))}{\d w^{(3)}} && \text{Chain Rule} \\ & = \frac{\d (y - f(x))^{2}}{\d(y - f(x))} \left(\frac{\d y}{\d w^{(3)}} - \frac{\d f(x)}{\d w^{(3)}} \right)\\ & = \frac{\d (y - f(x))^{2}}{\d(y - f(x))} \left(0 - \frac{\d f(x)}{\d w^{(3)}}\right) \\ & = - \frac{\d (y - f(x))^{2}}{\d(y - f(x))} \frac{\d f(x)}{\d w^{(3)}} \\ & = - \frac{\d (y - f(x))^{2}}{\d(y - f(x))} \frac{\d a^{(3)}}{\d w^{(3)}} \\ & = - \frac{\d (y - f(x))^{2}}{\d(y - f(x))} \frac{\d \sigma(z^{(3)})}{\d w^{(3)}} \\ & = - \frac{\d (y - f(x))^{2}}{\d(y - f(x))} \frac{\d \sigma(z^{(3)})}{\d z^{(3)}} \frac{\d z^{(3)}}{\d w^{(3)}} \\ & = - \frac{\d (y - f(x))^{2}}{\d(y - f(x))} \frac{\d \sigma(z^{(3)})}{\d z^{(3)}} \frac{\d (w^{(3)}a^{(2)} + b^{(3)})}{\d w^{(3)}} \\ \end{align*}

We have, \begin{align*} - \frac{\d (y - f(x))^{2}}{\d(y - f(x))} & = -2(y - f(x)) \\ \frac{\d \sigma(z^{(3)})}{\d z^{(3)}} & = \sigma'(z^{(3)}) \\ \frac{\d (w^{(3)}a^{(2)} + b^{(3)})}{\d w^{(3)}} &= \frac{\d (w^{(3)}a^{(2)})}{\d w^{(3)}} + \frac{\d (b^{(3)})}{\d w^{(3)}} \\ & = a^{(2)} + 0 \\ & = a^{(2)}\\ \end{align*}

We conclude that, $$\frac{\d C(x, f)}{\d w^{(3)}} = -2(y - f(x))\sigma'(z^{(3)})a^{(2)}$$

If we keep applying the chain chain rule in the above derivation, we can solve for the other partial derivatives as well and get,

\begin{align*} \frac{\d C(x, f)}{\d w^{(3)}} & = -2(y - f(x))\sigma'(z^{(3)})a^{(2)} \\ \frac{\d C(x, f)}{\d w^{(2)}} & = -2(y - f(x))\sigma'(z^{(3)})w^{(3)}\sigma'(z^{(2)})a^{(1)} \\ & = -2(y - f(x))\sigma'(z^{(3)})w^{(3)}\sigma'(z^{(2)})x \end{align*}

For the partial derivatives with partial biases, the same above derivation stands mutatis mutandis, i.e., towards the end we'll have

$$\frac{\d (b^{(3)})}{\d b^{(3)}} = 1 \Rightarrow \frac{\d C}{\d b^{(3)}} = -\frac{\d (y - a^{(3)})^2}{\d (y-a^{(3)})} \frac{\d (\sigma(z^{(3)}))}{\d z^{(3)}} \frac{\d z^{(3)}}{\d b^{(3)}} = -2(y-a^{(3)})\sigma'(z^{(3)})(1)$$

Therefore,

\begin{align*} \frac{\d C(x, f)}{\d b^{(3)}} & = -2(y - f(x))\sigma'(z^{(3)}) \\ \frac{\d C(x, f)}{\d b^{(2)}} & = -2(y - f(x))\sigma'(z^{(3)})w^{(3)}\sigma'(z^{(2)}) \\ \end{align*}

Now you may start at $$w^{(3)} = w^{(2)} = b^{(3)} = b^{(2)} = 0$$ and say we are trying to classify even numbers from odds and decide that for all even $$x, f(x) = 1$$. We check,

$$f(2) = \sigma(0 \times \sigma(0\times 2 + 0) + 0) = 0$$

Well, this is not correct so now evaluate $$\nabla C(2, f)$$ for $$f(x)=0$$ and $$y=1$$ and adjust the $$w^{(3)}, w^{(2)}, b^{(3)}, b^{(2)}$$. The new weights and biases are,

$$[w^{(3)}, b^{(3)}, w^{(2)}, b^{(2)}] + \eta\nabla C(2, f)$$

Where $$\eta \in [0, 1]$$ is usually called the learning rate. Now take your new weights and biases and check how close to the expected $$y$$ are you? Repeat the above step for a different even number till your weights and biases have converged to where $$f(x)$$ for even numbers is always 1. At that point your toy network has learned all it could.