Question about Neural Network Training

Question

What is the benefit of optimizing our neural network error function using back prop rather than just using gradient or stochastic gradient descent directly on the error function?

How come we don't just take the partial derivative of the error function with respect to every weight and do gradient or stochastic gradient descent based on that?

Instead of doing back propagation to come up with these artificial error values for hidden nodes and then for each hidden node multiplying its error times the various partial derivatives of weights associated to its incoming edges, as if that hidden node was actually an output node.

Answer 1

Backpropagation is basically using the chain rule to efficiently calculate the gradient of the error function with respect to the weights. You need to calculate those gradients for either stochastic gradient descent, conjugate gradients, or any other gradient based optimization method.

It is usually identified with gradient descent, but you can see it independently of it. It allows you to calculate gradients of the error function with respect to the weights with a complexity that scales linearly with the number of weights.