Also, some good insights from Ian Goodfellow
answering to why do not use the whole training set to compute the gradient? on Quora:
The size of the learning rate is limited mostly by factors like how
curved the cost function is. You can think of gradient descent as
making a linear approximation to the cost function, then moving
downhill along that approximate cost. If the cost function is highly
non-linear (highly curved) then the approximation will not be very
good for very far, so only small step sizes are safe. You can read
more about this in Chapter 4 of the deep learning textbook, on
numerical computation:
http://www.deeplearningbook.org/contents/numerical.html
When you put
m examples in a minibatch, you need to do O(m) computation and use
O(m) memory, but you reduce the amount of uncertainty in the gradient
by a factor of only O(sqrt(m)). In other words, there are diminishing
marginal returns to putting more examples in the minibatch. You can
read more about this in Chapter 8 of the deep learning textbook, on
optimization algorithms for deep learning:
http://www.deeplearningbook.org/contents/optimization.html
Also, if
you think about it, even using the entire training set doesn’t really
give you the true gradient. The true gradient would be the expected
gradient with the expectation taken over all possible examples,
weighted by the data generating distribution. Using the entire
training set is just using a very large minibatch size, where the size
of your minibatch is limited by the amount you spend on data
collection, rather than the amount you spend on computation.
