Is there any literature that examines the choice of minibatch size when performing stochastic gradient descent? In my experience, it seems to be an empirical choice, usually found via cross-validation or using varying rules of thumb.

Is it a good idea to slowly increase the minibatch size as validation error decreases? What effects would this have on generalization error? Am I better-off using an extremely small minibatch and updating my model hundreds of thousands of times? Would I be better off with a balanced number somewhere between extremely small, and batch?
Should I scale the size of my minibatch with the size of the dataset, or the expected number of features within the dataset?

I obviously have a lot of questions about implementing minibatch learning schemes. Unfortunately, most papers I read don't really specify how they chose this hyperparameter. I've had some success from authors such as Yann LeCun, especially from the Tricks of the Trade collection of papers. However, I still haven't seen these questions fully addressed. Does anyone have any recommendations for papers, or advice as to what criteria I can use to determine good minibatch sizes when trying to learn features?

  • $\begingroup$ I don't seem to be getting a lot of hits on this topic. Is there a better stack exchange site to be asking machine learning or deep learning questions such as this on? $\endgroup$ Aug 26, 2013 at 22:36
  • $\begingroup$ FYI: cs.ubc.ca/~mpf/2011-hybrid-for-data-fitting.html $\endgroup$
    – Memming
    Sep 11, 2013 at 16:17
  • $\begingroup$ In practice, the answer is "as many samples as you can cram into your GPUs". Which is often a single-digit number because networks are pretty big compared to your typical GPU. $\endgroup$
    – Navin
    Dec 14, 2021 at 2:30

1 Answer 1


The theory for the effectiveness of SGD was worked out on single example updates (i.e. minibatch size 1), so using larger minibatches isn't theoretically necessary. It has two practical advantages:

One, if the computation can be vectorized, you might be able to compute gradients for a small minibatch >1 nearly equally as quickly, leading to significant speed increases in training.

In this case, the optimal minibatch size is a function of the particular hardware and implementation you're working with, so you're probably best off experimenting to find the sweet spot.

Two, computing the gradient on a minibatch size >1 will lead to more accurate gradients and more optimal steps. But this benefit will arrive and level off quickly once the minibatch size is increased beyond 1, so you can focus primarily on the first objective.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.