2
$\begingroup$

Is stochastic gradient descent basically the name given to mini-batch training where batch size = 1 and selecting random training rows? i.e. it is the same as 'normal' gradient descent, it's just the manner in which the training data is supplied that makes the difference?

One thing that confuses me is I've seen people say that even with SGD you can supply more than 1 data point, and have larger batches, so won't that just make it 'normal' mini-batch gradient descent?

$\endgroup$
2
$\begingroup$

Standard gradient descent and batch gradient descent were originally used to describe taking the gradient over all data points, and by some definitions, mini-batch corresponds to taking a small number of data points (the mini-batch size) to approximate the gradient in each iteration. Then officially, stochastic gradient descent is the case where the mini-batch size is 1.

However, perhaps in an attempt to not use the clunky term "mini-batch", stochastic gradient descent almost always actually refers to mini-batch gradient descent, and we talk about the "batch-size" to refer to the mini-batch size. Gradient descent with > 1 batch size is still stochastic, so I think it's not an unreasonable renaming, and pretty much no one uses true SGD with a batch size of 1, so nothing of value was lost.

$\endgroup$
  • $\begingroup$ Thanks @shimao. I just find it odd that I find information from the same author/publisher who have published information about both SGD and mini-batch GD as separate entities, when by all accounts, it's the same thing. But thanks for clarifying. $\endgroup$ – BigBadMe Mar 30 '18 at 19:36
2
$\begingroup$

Is stochastic gradient descent basically the name given to mini-batch training where batch size = 1 and selecting random training rows?

Yes. Though shuffling the rows isn't necessarily implied.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.