I'm currently taking Andrew Ng's Machine Learning Coursera course, and I'm not sure that I fully understand the Delta gradients in Backpropagation (BP).

I see that for the first layer of BP (which is really the output layer of the network) we can calculate the gradient by subtracting the real output label vector from the output vector generated by the network. However, there is a lot of reference to these delta terms as gradients and partial derivatives. Specifically, he says:

I'm not sure why we are computing derivatives here, and moreover why computing consecutive derivatives for each layer results in overall better weights (especially if we end up just adding them all to some "accumulator":

Any insight would be awesome.

A link to a one page PDF summery of the lesson is here, if it helps: https://www.coursera.org/learn/machine-learning/supplement/pjdBA/backpropagation-algorithm

• Are you familiar with gradient descent? – Matthew Drury Jul 27 '17 at 4:56
• @MatthewDrury Yes, the gradient descent algorithm makes much more sense to me for whatever reason. – Sam Jul 27 '17 at 5:37
• Backpropagation is just gradient descent with some clever tricks to help computation. – Matthew Drury Jul 27 '17 at 5:51