TL;DR: Too large a mini-batch size usually leads to a lower accuracy!
For those interested, here's an explanation.
There are two notions of speed:
- Computational speed
- Speed of convergence of an algorithm
Computational speed is simply the speed of performing numerical calculations in hardware. As you said, it is usually higher with a larger mini-batch size. That's because linear algebra libraries use vectorization for vector and matrix operations to speed them up, at the expense of using more memory. Gains can be significant up to a point. From my experience, there is a point after which there are only marginal gains in speed, if any. The point depends on the data set, hardware, and a library that's used for numerical computations (under the hood).
But, let's not forget that there is also the other notion of speed, which tells us how quickly our algorithm converges.
Firstly, what does it mean for our algorithm to converge? Well, it's up to us to define and decide when we are satisfied with an accuracy, or an error, that we get, calculated on the validation set. We can either define it in advance and wait for the algorithm to come to that point, or we can monitor the training process and decide to stop it when the validation error starts to rise significantly (the model starts to overfit the data set). We really shouldn't stop it right away, the first moment the error starts to rise, if we work with mini batches, because we use Stochastic Gradient Descent, SGD. In case of (full batch) Gradient Descent, after each epoch, the algorithm will settle in a minimum, be it a local or the global one. SGD never really settles in a minimum. It keeps oscillating around it. It could go on indefinitely, but it doesn't matter much, because it's close to it anyway, so the chosen values of parameters are okay, and lead to an error not far away from the one found at the minimum.
Now, after all that theory, there's a "catch" that we need to pay attention to. When using a smaller batch size, calculation of the error has more noise than when we use a larger batch size. One would say, well, that's bad, isn't it? The thing is, that noise can help the algorithm jump out of a bad local minimum and have more chance of finding either a better local minimum, or hopefully the global minimum.
Thus, if we can find a better solution more quickly by using a smaller batch size instead of a larger one, just by the help of the "unwanted" noise, we can tune between the total time it takes for our algorithm to find a satisfactory solution and a higher accuracy.
What I want to say is, for a given accuracy (or error), smaller batch size may lead to a shorter total training time, not longer, as many believe.
Or, if we decide to keep the same training time as before, we might get a slightly higher accuracy with a smaller batch size, and we most probably will, especially if we have chosen our learning rate appropriately.
If you have time, check out this paper:
Systematic evaluation of CNN advances on the ImageNet
Especially, check out "3.7. Batch size and learning rate", and Figure 8.
You will see that large mini-batch sizes lead to a worse accuracy, even if tuning learning rate to a heuristic.
In general, batch size of 32 is a good starting point, and you should also try with 64, 128, and 256. Other values (lower or higher) may be fine for some data sets, but the given range is generally the best to start experimenting with. Though, under 32, it might get too slow because of significantly lower computational speed, because of not exploiting vectorization to the full extent. If you get an "out of memory" error, you should try reducing the mini-batch size anyway.
So, it's not simply about using the largest possible mini-batch size that fits into memory.
To conclude, and answer your question, a smaller mini-batch size (not too small) usually leads not only to a smaller number of iterations of a training algorithm, than a large batch size, but also to a higher accuracy overall, i.e, a neural network that performs better, in the same amount of training time, or less.
Don't forget that the higher noise can help it jump out of a bad local minimum, rather than leaving it stuck in it.