Batch gradient descent refers to accumulating the loss (and gradients) over all
n training examples and then performing a single step.
In Sebastian Ruder's post on optimization, he writes
"batch gradient descent can be very slow and is intractable for datasets that don't fit in memory".
I understrand the inefficiency of batch gradient descent, but why does it require storing the entire datset in memory? ostensibly, a given example must be saved in memory only when iterating over it (and calculating its loss). When continuing to the next example, we can cast aside the previous one.
Is it somehow related to the structure of the computation graph that automatic differentiation libraries build for batched data?