This is a pretty basic question.

So...backprop is an efficient algorithm for computing the gradients used by the optimizer to improve model parameters, no matter if SDG or something else. I get that.

The actual difference between classic gradient descent and stochastic gradient descent is the batchsize used for computing the gradients, thats why SGD is more efficient. I get that as well.

But if I now use backprop....where is then the difference between them? We move in the direction of the negative gradient, that holds for both of them. So again where is the difference?

Edit: To prevent misunderstanding. The difference between SGD and GD after use of backprop is meant, not the difference between backprop and SGD/GD.

Thanks for the contributions. The differentiation between backprop plus optimization and the learning process as a whole, which itself is also often called backprop, was the reason for my question. It implied for me, that if the backprop computes the gradients and the optimizer only modifies the parameters afterwards, that there had to be a difference in the way they do it except for the different gradients.

  • $\begingroup$ backpropagation is how one determines the gradient at a location in a neural network. One forward propagates the input to get the estimated output, then from the target computes the error, then reverse propagates the error through the network to determine what the relationship is between the error and weights. Then using momentum, and learning rates, and even random sampling, one can use sequential measurements of the error values along with these transformation strategies to reduce the ensemble error summary statistic. $\endgroup$ – EngrStudent Jan 12 at 19:48

The core concept is that the gradient is a statistic, a piece of information estimated from a limited sample.

The difference between GD and SGD is that if you repeated SGD twice for the same initial parameter values but use different batches, you're likely to get a different estimate of the gradient. This is because the SGD gradient is computed with respect to the loss function computed using the random selection of observations used in the mini-batch. Using a different mini-batch implies different feature values, different target values, different loss estimates, and therefore different gradient estimates. (While a person could contrive a scenario where a specific model with two well-chosen mini-batches would have the same gradient but different features and target values could have the same estimated gradient, but this is a special case and not germane to the motivation of SGD.)

But if the gradient is computed using the same data (such as always using all of the samples), then clearly there's no stochastic component, because the data are the same in both instances. Computing the gradient twice for the same parameter values for the same data values will yield the same result.


We move in the direction of the negative gradient, but the gradient is different, because in (full-batch) GD and in (batch) SGD the data are different!

And that's the point: SGD adds randomness so that it can more easily escape local minima.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.