Batch gradient descent refers to accumulating the loss (and gradients) over all n training examples and then performing a single step.

In Sebastian Ruder's post on optimization, he writes

"batch gradient descent can be very slow and is intractable for datasets that don't fit in memory".

I understrand the inefficiency of batch gradient descent, but why does it require storing the entire datset in memory? ostensibly, a given example must be saved in memory only when iterating over it (and calculating its loss). When continuing to the next example, we can cast aside the previous one.

Is it somehow related to the structure of the computation graph that automatic differentiation libraries build for batched data?


but why does it require storing the entire datset in memory?

It doesn't. You could potentially load small chunks into memory, calculate their loss, and then sum it up to loss value. The problem is that would be slow, because you'd need to load data to memory multiple times before making updates.

  • $\begingroup$ I'm not sure I buy this: the ratio of disk time to compute time is the same between batch GD and SGD, so why is loading data to memory not a problem for SGD? $\endgroup$ – shimao May 17 '18 at 20:31
  • $\begingroup$ You are correct. I didn't say this is a problem - only that this is slow compared to using gradient estimate from minibatches. $\endgroup$ – Jakub Bartczuk May 17 '18 at 20:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.