I'm new to machine learning, and am confused about some aspects of stochastic gradient decent.
I've read in several places that, when using vectorized code, the reason that mini-batching in stochastic gradient descent is necessary is to avoid performance degradation when memory limits are hit for too-large mini-batches. But in my code, long before I see any such degradation in processing rate, s/epoch, I see a completely different effect: dramatic decreases in learning rate.
This is no surprise, since for nearly flat processing rate gradient descent rates will be inversely proportional to batch size: s/mini-bach must be increasing. And since (for fixed $\eta$) it is this rate that determines the learning rate, larger mini-batches must — barring some dramatic unexpected and unobserved increase in processing rate with mini-batch size — always be worse than that for smaller batches.
(Moreover any per-epoch analysis of preformance, especially of the training set, will add additional time that grows with mini-batch size.)
It seems then that choosing a mini-batch size depends entirely on selecting a value large enough for vectorization to "kick in" (very small batches perform poorly in that regard) and not going any further than that, except to reduce learning noise.
Is this, in fact, how one chooses a batch size in practice: by picking the lowest possible "well behaved" batch size (e.g. perhaps where the "level off" point is for processing rate) for one’s algorithm, and using that for all networks and training sets? If so, will that number remain the same for my code on all hardware, or is it an unpredictable function of hardware and software? And — more importantly — what of the story that the reason for mini-batching is to avoid memory issues?
This is consistent with what I see across the board, for both training and test accuracies, for a range of network sizes (the number is the number nodes in the hidden layer of a simple 3-layer "beginner" network):