# Does memory ever really matter for mini-batch size selection?

I'm new to machine learning, and am confused about some aspects of stochastic gradient decent.

I've read in several places that, when using vectorized code, the reason that mini-batching in stochastic gradient descent is necessary is to avoid performance degradation when memory limits are hit for too-large mini-batches. But in my code, long before I see any such degradation in processing rate, s/epoch, I see a completely different effect: dramatic decreases in learning rate.

This is no surprise, since for nearly flat processing rate gradient descent rates will be inversely proportional to batch size: s/mini-bach must be increasing. And since (for fixed $\eta$) it is this rate that determines the learning rate, larger mini-batches must — barring some dramatic unexpected and unobserved increase in processing rate with mini-batch size — always be worse than that for smaller batches.

(Moreover any per-epoch analysis of preformance, especially of the training set, will add additional time that grows with mini-batch size.)

It seems then that choosing a mini-batch size depends entirely on selecting a value large enough for vectorization to "kick in" (very small batches perform poorly in that regard) and not going any further than that, except to reduce learning noise.

Is this, in fact, how one chooses a batch size in practice: by picking the lowest possible "well behaved" batch size (e.g. perhaps where the "level off" point is for processing rate) for one’s algorithm, and using that for all networks and training sets? If so, will that number remain the same for my code on all hardware, or is it an unpredictable function of hardware and software? And — more importantly — what of the story that the reason for mini-batching is to avoid memory issues?

This is consistent with what I see across the board, for both training and test accuracies, for a range of network sizes (the number is the number nodes in the hidden layer of a simple 3-layer "beginner" network):

• I would add one observation. You say "other than to reduce learning noise". I believe this is not a factor at all. If you split a batch into two smaller batches and lower the learning rate appropriately, you will on average get slightly better learning from the two batches: this principle extends right down to single sample batches. It is only the advantage of faster processing in parallel outweighing this that makes it useful to use minibatches of size greater than 1. – Elroch May 25 '18 at 12:33