I'm following Geoffrey Hinton's Neural Networks Lectures (this one in particular: https://www.youtube.com/watch?v=LOc_y67AzCA), and I'm stumped by the second equation here:

backprop dE/dy

As I understand it, $E = \frac{1}{2} \sum\limits_n (t_n - y_n)^2$ ranges over all neurons in the network, and all neurons above $y_i$ depend on $y_i$. Then shouldn't that mean $\frac{\partial E}{\partial y_i} = \sum\limits_n \frac{\partial E}{\partial z_n} \frac{\partial z_n}{\partial y_i}$ where $n$ ranges over all neurons (above layer $i$)? But the way the second equation in the picture indexes over $j$ makes it look like it's only over the layer $j$ above $y_i$. What about the other neurons in layer $j + 1, j + 2,...$?


Ah I understand now. Thanks to Rabindra for the hint. I think this is how it goes:

At layer $i$, $E$ is considered a function of the $z_j$'s in layer $j$, and the $z_j$'s depend on $y_i$. So by the chain rule we have the second equation: $$\frac{\partial E}{\partial y_i} = \sum\limits_j \frac{\partial E}{\partial z_j} \frac{\partial z_j}{\partial y_i} = \sum\limits_j w_{ij} \frac{\partial E}{\partial z_j}.$$ But we don't know $\frac{\partial E}{\partial z_j},$ so we use the first equation: $$\frac{\partial E}{\partial z_j} = \frac{\partial E}{\partial y_j} \frac{d y_j}{d z_j} = y_j (1 - y_j) \frac{\partial E}{\partial y_j}.$$ Here we use the second equation again to go up another layer, and so on. (In practice you'd start from the top and calculate down.)

Along the way down we use the third equation to calculate $\frac{\partial E}{\partial w_{ij}},$ and once we have all of those we use the Delta Rule to calculate the necessary weight changes $\Delta w_{ij} = - \alpha \frac{\partial E}{\partial w_{ij}}.$


1 Answer 1


You are backpropagating the error layer by layer, so before you calculate $\frac{\partial E}{\partial y_i} $, $\frac{\partial E}{\partial y_j}$ must have been calculated. If there is a layer j+1(let's denote as k), then $\frac{\partial E}{\partial y_j} = \sum\limits_k \frac{\partial E}{\partial z_k} \frac{\partial z_k}{\partial y_j}$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.