0
$\begingroup$

What is the benefit of optimizing our neural network error function using back prop rather than just using gradient or stochastic gradient descent directly on the error function?

How come we don't just take the partial derivative of the error function with respect to every weight and do gradient or stochastic gradient descent based on that?

Instead of doing back propagation to come up with these artificial error values for hidden nodes and then for each hidden node multiplying its error times the various partial derivatives of weights associated to its incoming edges, as if that hidden node was actually an output node.

$\endgroup$
2
$\begingroup$

Backpropagation is basically using the chain rule to efficiently calculate the gradient of the error function with respect to the weights. You need to calculate those gradients for either stochastic gradient descent, conjugate gradients, or any other gradient based optimization method.

It is usually identified with gradient descent, but you can see it independently of it. It allows you to calculate gradients of the error function with respect to the weights with a complexity that scales linearly with the number of weights.

$\endgroup$
  • $\begingroup$ So what you're saying is that back prop is just a really clever way of doing gradient descent which takes advantage of the fact that the network function has a nice form (it's essentially the composition of the same type of function over and over again) in order to maintain efficiency for a large number of hidden nodes/layers? $\endgroup$ – Thoth Jul 14 '14 at 0:05
  • $\begingroup$ And so if one wanted to they could train the network using regular gradient descent rather than back prop and it would optimize in the same exact way, except it would be much less efficient? $\endgroup$ – Thoth Jul 14 '14 at 0:07
  • $\begingroup$ Backpropagation is a clever way to calculate the gradient of the error function with respect to the weights. You can then use conjugate gradients, BFGS, or any other optimization method you prefer. $\endgroup$ – jpmuc Jul 14 '14 at 7:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.