Choosing an appropriate minibatch size for stochastic gradient descent (SGD)

Question

Is there any literature that examines the choice of minibatch size when performing stochastic gradient descent? In my experience, it seems to be an empirical choice, usually found via cross-validation or using varying rules of thumb.

Is it a good idea to slowly increase the minibatch size as validation error decreases? What effects would this have on generalization error? Am I better-off using an extremely small minibatch and updating my model hundreds of thousands of times? Would I be better off with a balanced number somewhere between extremely small, and batch?
Should I scale the size of my minibatch with the size of the dataset, or the expected number of features within the dataset?

I obviously have a lot of questions about implementing minibatch learning schemes. Unfortunately, most papers I read don't really specify how they chose this hyperparameter. I've had some success from authors such as Yann LeCun, especially from the Tricks of the Trade collection of papers. However, I still haven't seen these questions fully addressed. Does anyone have any recommendations for papers, or advice as to what criteria I can use to determine good minibatch sizes when trying to learn features?

Answer 1

The theory for the effectiveness of SGD was worked out on single example updates (i.e. minibatch size 1), so using larger minibatches isn't theoretically necessary. It has two practical advantages:

One, if the computation can be vectorized, you might be able to compute gradients for a small minibatch >1 nearly equally as quickly, leading to significant speed increases in training.

In this case, the optimal minibatch size is a function of the particular hardware and implementation you're working with, so you're probably best off experimenting to find the sweet spot.

Two, computing the gradient on a minibatch size >1 will lead to more accurate gradients and more optimal steps. But this benefit will arrive and level off quickly once the minibatch size is increased beyond 1, so you can focus primarily on the first objective.