Is there any literature that examines the choice of minibatch size when performing stochastic gradient descent? In my experience, it seems to be an empirical choice, usually found via cross-validation or using varying rules of thumb.
Is it a good idea to slowly increase the minibatch size as validation error decreases? What effects would this have on generalization error?
Am I better-off using an extremely small minibatch and updating my model hundreds of thousands of times? Would I be better off with a balanced number somewhere between extremely small, and batch?
Should I scale the size of my minibatch with the size of the dataset, or the expected number of features within the dataset?
I obviously have a lot of questions about implementing minibatch learning schemes. Unfortunately, most papers I read don't really specify how they chose this hyperparameter. I've had some success from authors such as Yann LeCun, especially from the Tricks of the Trade collection of papers. However, I still haven't seen these questions fully addressed. Does anyone have any recommendations for papers, or advice as to what criteria I can use to determine good minibatch sizes when trying to learn features?