1
$\begingroup$

I've been working on trying to understand the backpropogation algorithm and the calculus behind it, and in my work I have stumbled across a sort of odd situation. I am just practicing on a 1 input, 1 output, 1 hidden node network, and after taking the derivatives of the second weight in the network in the connection from the hidden to the out put node, I got this...

I am using a simple quadratic cost function of $C(a) = (a^l-y)^2$ where a is the activation and l is used to indicate which layer relative to the output layer. I am using the sigmoid function for my non-linearity. So from that I have $a^l(z^l) = \sigma(z^l)$ and for z $z^l(w^l) = a^{(l-1)}*w^l$ and so, using the chain rule, $\frac{\partial C}{\partial w^l} = \frac{\partial C}{\partial a^l} * \frac{\partial a^l}{\partial z^l} * \frac{\partial z^l}{\partial w^l} = 2(a^l-y)*(\sigma(z^l)*(1-\sigma(z^l)))*a^{l-1}$

Now I may have done this wrong, but I am fairly confident in my work here. So I have defined the variables above so that $a^l = 1, y = 0,z^l=50,C=1,w^l=0.8,a^{l-1}=62.5$ SO clearly there is work to be done on this weight, but if you plug each value into the derivation above you get 0 due to the derivation a with respect to z and the sigmoid function derivation. This makes no sense, there is clearly a negative gradient do be added but it comes out to be zero why is that?

Thanks is advance, additionally if anyone has some resources that may better help me understand it would be greatly appreciated if you could share them here

** Note the notation is probably not perfect, so feel free to change it.

$\endgroup$
1
$\begingroup$

Actually, $$a^l=\sigma(z^l)=\frac{1}{1+\exp(-z^l)}=\frac{1}{1+\exp(-50)}\neq 1$$

but since $\exp(-50)$ is a very small number, it underflows and the result can be $1$ in finite precision. So, This is more of a numerical issue than theoretical. You should normalize your inputs to not come across these types of problems.

P.S. it should be $\sigma(1-\sigma)$, not $\sigma-(1-\sigma)$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.