In tutorials, gradient descent is often shown as a point descending down a bowl shaped error surface. As it learns from examples it descends down the surface towards the minima.

It struck me today though that the error surface is defined in a nueral network per training example. So the surface should appear different for each example. Why then do I never see the error surface morphing as the point descends downwards?

Has my intuition lost it's way somewhere?

In normal gradient descent, the error surface is for all the examples. So it's a sum of the error surfaces for each example.

Stochastic gradient descent uses a subset of examples for each step. So in that case, the error surface is different for each step, but tends (in some sense), to the global error surface as the number of steps increases.

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.