# Backpropagation algorithm in neural networks (NN) with logistic activation function

In this Coursera course by Geoffrey Hinton, the backpropagation algorithm is described starting at min 8 of this video, and when completed it looks like this: The slides can be found here.

Now, the critical value to assess is $\color{blue}{\frac{\partial E}{\partial w_{ij}}}$, which relates the changes in the error in the training set $(E)$ to the set of weights $(w_{ij})$.

Working through the equations, $\color{blue}{\frac{\partial E}{\partial w_{ij}}}$ depends on $\color{red}{\frac{\partial E}{\partial z_j}}$:

$$\color{blue}{\frac{\partial E}{\partial w_{ij}}}= y_i\,\color{red}{\frac{\partial E}{\partial z_j}}$$

We find $\color{red}{\frac{\partial E}{\partial z_j}}$ in the first equation:

$$\color{red}{\frac{\partial E}{\partial z_j}}=y_j\,(1-y_j)\,\color{orange}{\frac{\partial E}{\partial y_j}}.$$

Unfortunately, it feels as though we get into a loop on the second equation, where this latter partial of $E$, the expression $\color{orange}{\frac{\partial E}{\partial y_j}}$ seems to recursively refer us back to $\color{red}{\frac{\partial E}{\partial z_j}}:$

$$\color{orange}{\frac{\partial E}{\partial y_j}}=\sum_j w_{ij}\color{red}{\frac{\partial E}{\partial z_j}}$$

What am I missing? What is the right way of walking though the three equations on the posted image?

EDIT:

After the comment regarding the last layer being simply the partial derivative of the loss function, is it as follows:

$$\frac{\partial E}{\partial y_i}=\frac{\partial \frac{1}{2}(y-y_i)^2}{\partial y_i}=y_i-y$$

?

• Notice that in the last layer $\frac{\partial E}{\partial y_i}$ doesn't depend on $\frac{\partial E}{\partial z_j}$ and is simply the partial derivative of loss function so we break the recurence. – Łukasz Grad Mar 14 '17 at 23:32
• @ŁukaszGrad I edited the OP with what I got from your comment. It is possibly now a yes or no type of answer, and I wonder if you can please take a look. ty – Antoni Parellada Mar 15 '17 at 14:54

Just to add (sorry for being nitpicky :), when you write $\frac{\partial E}{\partial y_i}$ it is implied that the output is a vector, so maybe writing
$$\frac{\partial\frac{1}{2}\sum_i(t_i - y_i)^2}{\partial y_i} = \frac{\partial\frac{1}{2}(t_i - y_i)^2}{\partial y_i} = y_i - t_i$$
would be more clear, for $T = (t_1, \dots, t_n)^T$ being the correct output.
• I don't see the $T$ in the equation. – Antoni Parellada Mar 15 '17 at 16:09
• It is the same as $y$ in your equation, that is $y = t_i$ for some $i$ here, maybe you meant scalar output? Then it would just be $T = (t_1) = y$ (which is exactly what you wrote) – Łukasz Grad Mar 15 '17 at 16:12