6
$\begingroup$

In this Coursera course by Geoffrey Hinton, the backpropagation algorithm is described starting at min 8 of this video, and when completed it looks like this:

enter image description here

The slides can be found here.

Now, the critical value to assess is $\color{blue}{\frac{\partial E}{\partial w_{ij}}}$, which relates the changes in the error in the training set $(E)$ to the set of weights $(w_{ij})$.

Working through the equations, $\color{blue}{\frac{\partial E}{\partial w_{ij}}}$ depends on $\color{red}{\frac{\partial E}{\partial z_j}}$:

$$\color{blue}{\frac{\partial E}{\partial w_{ij}}}= y_i\,\color{red}{\frac{\partial E}{\partial z_j}}$$

We find $\color{red}{\frac{\partial E}{\partial z_j}}$ in the first equation:

$$\color{red}{\frac{\partial E}{\partial z_j}}=y_j\,(1-y_j)\,\color{orange}{\frac{\partial E}{\partial y_j}}.$$

Unfortunately, it feels as though we get into a loop on the second equation, where this latter partial of $E$, the expression $\color{orange}{\frac{\partial E}{\partial y_j}}$ seems to recursively refer us back to $\color{red}{\frac{\partial E}{\partial z_j}}:$

$$\color{orange}{\frac{\partial E}{\partial y_j}}=\sum_j w_{ij}\color{red}{\frac{\partial E}{\partial z_j}}$$

What am I missing? What is the right way of walking though the three equations on the posted image?


EDIT:

After the comment regarding the last layer being simply the partial derivative of the loss function, is it as follows:

$$\frac{\partial E}{\partial y_i}=\frac{\partial \frac{1}{2}(y-y_i)^2}{\partial y_i}=y_i-y$$

?

$\endgroup$
2
  • 1
    $\begingroup$ Notice that in the last layer $\frac{\partial E}{\partial y_i}$ doesn't depend on $\frac{\partial E}{\partial z_j}$ and is simply the partial derivative of loss function so we break the recurence. $\endgroup$ Mar 14, 2017 at 23:32
  • $\begingroup$ @ŁukaszGrad I edited the OP with what I got from your comment. It is possibly now a yes or no type of answer, and I wonder if you can please take a look. ty $\endgroup$ Mar 15, 2017 at 14:54

1 Answer 1

4
$\begingroup$

Yes you got it right.

Just to add (sorry for being nitpicky :), when you write $\frac{\partial E}{\partial y_i}$ it is implied that the output is a vector, so maybe writing

$$\frac{\partial\frac{1}{2}\sum_i(t_i - y_i)^2}{\partial y_i} = \frac{\partial\frac{1}{2}(t_i - y_i)^2}{\partial y_i} = y_i - t_i$$

would be more clear, for $T = (t_1, \dots, t_n)^T$ being the correct output.

$\endgroup$
2
  • $\begingroup$ I don't see the $T$ in the equation. $\endgroup$ Mar 15, 2017 at 16:09
  • $\begingroup$ It is the same as $y$ in your equation, that is $y = t_i$ for some $i$ here, maybe you meant scalar output? Then it would just be $T = (t_1) = y$ (which is exactly what you wrote) $\endgroup$ Mar 15, 2017 at 16:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.