13
$\begingroup$

Is there any literature that examines the choice of minibatch size when performing stochastic gradient descent? In my experience, it seems to be an empirical choice, usually found via cross-validation or using varying rules of thumb.

Is it a good idea to slowly increase the minibatch size as validation error decreases? What effects would this have on generalization error? Am I better-off using an extremely small minibatch and updating my model hundreds of thousands of times? Would I be better off with a balanced number somewhere between extremely small, and batch?
Should I scale the size of my minibatch with the size of the dataset, or the expected number of features within the dataset?

I obviously have a lot of questions about implementing minibatch learning schemes. Unfortunately, most papers I read don't really specify how they chose this hyperparameter. I've had some success from authors such as Yann LeCun, especially from the Tricks of the Trade collection of papers. However, I still haven't seen these questions fully addressed. Does anyone have any recommendations for papers, or advice as to what criteria I can use to determine good minibatch sizes when trying to learn features?

$\endgroup$
  • $\begingroup$ I don't seem to be getting a lot of hits on this topic. Is there a better stack exchange site to be asking machine learning or deep learning questions such as this on? $\endgroup$ – Jason_L_Bens Aug 26 '13 at 22:36
  • $\begingroup$ FYI: cs.ubc.ca/~mpf/2011-hybrid-for-data-fitting.html $\endgroup$ – Memming Sep 11 '13 at 16:17
6
$\begingroup$

The theory for the effectiveness of SGD was worked out on single example updates (i.e. minibatch size 1), so using larger minibatches isn't theoretically necessary. It has two practical advantages:

One, if the computation can be vectorized, you might be able to compute gradients for a small minibatch >1 nearly equally as quickly, leading to significant speed increases in training.

In this case, the optimal minibatch size is a function of the particular hardware and implementation you're working with, so you're probably best off experimenting to find the sweet spot.

Two, computing the gradient on a minibatch size >1 will lead to more accurate gradients and more optimal steps. But this benefit will arrive and level off quickly once the minibatch size is increased beyond 1, so you can focus primarily on the first objective.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.