Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I want to understand why mini-batching is used to train neural networks (rather than using the entire dataset for every update).
Is the reason purely that with big datasets, it requires big computing power to use entire dataset every time, so mini-batching is the intermediate solution?
If so, does that mean that it's best to use the biggest batch size your machine can handle when training neural networks?
Any insight, references or info would be appreciated!
There is evidence that supports the proposition that it is best to use the biggest batch size your machine can handle. See e.g. Goyal et al. (2018).
However, that paper (and another) reveal optimization difficulties with extremely large batch sizes. There is concern that large-batch optimization converges to "sharp" minima that generalize less well. Goyal et al. present some strategies to avoid those optimization difficulties (e.g. low-training-rate warmup phase, how to adjust learning rate dependent on batch size, training data shuffling each epoch).
It's complicated. We're trying to balance multiple effects. Yes, you are right in your reasoning as far as it goes, but there are other considerations as well.
GPU utilization favors larger batches, up to a point. Larger batches allow us to parallelize the computation -- up until you max out the capacity of your GPU. GPUs are designed to do many copies of the same computation in parallel. So, a mini-batch of only one sample is usually wasteful: it doesn't use the full capacity of your GPU. Depending on your GPU and the network, there will be some number of copies of the neural network it can run in parallel. It can be beneficial to set a batch size around that number. For instance, suppose your GPU can support a batch size of up to 64. Then using a batch size of 32 will only use 50% of the capacity of your GPU, wasting its power and causing training to take 2x longer than necessary; using a batch size of 64 will use its full capability and make training time the fastest possible; and using a batch size larger than 64 won't offer any further speedups.
Mathematical optimization favors larger batches, in some sense. One way to view training a neural network is as solving a mathematical optimization problem: namely, we're trying to find network weights that minimize the training loss. Gradient descent gives you some kind of approximation to the optimal solution. From that perspective, yes, if you fix the number of iterations of gradient descent, then the larger the batch size, the better the quality of the solution gradient descent finds (the lower the training loss). So, yes, from this perspective, what you write is absolutely correct.
Generalization favors smaller batches. It turns out there is another, subtler issue: generalization. It turns out that if you try as hard as you possibly can to find the optimal solution to the learning optimization problem (and in particular, run gradient descent for as long as you can stand to), neural networks tend to overfit to the training data and thus generalize poorly to other data. This is bad. There are various methods for combating overfitting, but one of the most fundamental methods used in neural networks is "early stopping": namely, we stop gradient descent early, before it has reached an optimal solution to the optimization problem. This is bit counter-intuitive, because it means that we are deliberately accepting a poorer solution to the optimization problem. It is surprising that it is beneficial to do so, but it turns out that early stopping acts as a form of regularization and helps the neural network generalize better. As far as I know it is an open question exactly why early stopping helps, but it does.
And, as a result, using very large batches is in tension with early stopping. My understanding is that using very large batches has a somewhat similar effect to running gradient descent for a long time, and thus may cause overfitting. Intuitively, running for 1000 iterations of gradient descent with a batch size of 128 is vaguely comparable to 2000 iterations with a batch size of 64, so you can perhaps see why, for a fixed number of iterations of gradient descent, larger batch sizes pose a greater risk of overfitting.
Just to add to the complications, in practice we don't hold the number of iterations fixed. Instead, we hold fixed the batch size * the number of iterations. This is typically measured in terms of the number of epochs, which is measured as the batch size * the number of iterations / the size of the training set. For a fixed number of epochs, it turns out that using a smaller batch size gives a slightly better solution to the optimization problem, so we have some incentives to use smaller batch sizes. This effect might be relatively minor, for the ranges of batch sizes used in practice, but if you tried to put the entire training set in one batch, this might potentially lead to a degradation in the effectiveness of the learned model. (The number of iterations determines how long it takes to train the model on a single GPU, in wall-clock time; the number of epochs determines how much energy or computation it takes to train the model, in total.)
Summary. We are dealing with competing concerns. From the perspective of generalization and overfitting, it might be best to use a batch size of 1, or small batches. But this would make very inefficient use of GPUs, and would make training take prohibitively long. From the perspective of using your GPU fully and making training complete in a reasonable amount of time, we are incentivized to use a batch size large enough to fully utilize all of the GPU's parallel computing units. So, in practice, it's common to choose batch sizes somewhere around there.
so you have to understand that neural networks are used as 'computation-bound' statistical models. ie they are solving the problem how accurate can I get with a budget of X compute-hours.
reasoning with a mental model of modelling in under an hour will lead you astray.
minibatch is a tradeoff of stochasticity and iteration speed vs accuracy. you will be able to achieve more iterations in the same time if you have a smaller minibatch - iteration speed
(batch) gradient descent will get stuck in local minima. so some stochasticity is beneficial.
stochastic gradient descent (ie updating after each sample) will be too noisy (ie the single sample gradient is too far from the batch gradient)
somewhere in between is therefore beneficial. the law of large numbers would suggest that accuracy of the estimate only scales with square root of the number of samples - halving the error requires 4 times the samples, quartering the error requires 16 times the samples (and therefore corresponding computation time...)
It might be useful to consider the extreme cases of mini-batch sizes: using a single sample vs taking all $N$ samples in one go. These are sometimes also referred to as on-line and batch learning, respectively.
For on-line learning (single sample), we could list the following properties (I am probably ignoring a lot here):
For batch learning ($N$ samples), on the other hand, we have
Conceptually, the on-line learning scenario is arguably more attractive. After all, we get much more updates out of a single pass through the data and generalisation performance should be better. The batch learning scenario, on the other hand, looks more promising in practice, because it can benefit from parallellisation, reducing run/waiting times. However, modern datasets can be so large that it has become impossible to use all data at once, which obviously cripples this argument.
In the end, the size of your mini-batches seems to be a trade-off between how long you want to wait for computations and how good you want your model to generalise. At least for small datasets.
If you are working with huge datasets, you will probably end up with mini-batches that consist of a collection of single samples for a subset of all possible classes. As a result, the gradients will be stochastic enough to provide generalisation benefits. Moreover, you might have to wait weeks instead of days if you don't take a sufficiently large batch size to finish a single pass through the data. Also, some architectures rely on a sufficiently large batch size to work properly (e.g. batchnorm, CLIP, ...)
TL;DR: it's a trade-off that probably depends on the size of the data.
Wait a minute... Who said that more updates is better?
That turns out to depend on how you do the updates. If the gradient for a single sample, $(\boldsymbol{x}_i, \boldsymbol{y}_i),$ would be $$\nabla_\theta L(g(\boldsymbol{x}_i \mathbin{;} \theta), \boldsymbol{y}_i)$$ (with $\theta$ the set of parameters and $L$ some loss function), the gradient for the entire dataset should equal $$\sum_{i=1}^N \nabla_\theta L(g(\boldsymbol{x}_i \mathbin{;} \theta), \boldsymbol{y}_i).$$
If we use each sample exactly once in the on-line setting the $N$ updates should be roughly equivalent to the single update from the full-batch setting. Especially if the learning rate is small enough (i.e. the individual updates do not change the weights too much).
The figure below illustrates the learning curves when trained on the sum of errors.
Note that performance does not depend on the number of updates (but it does depend on the number of passes through the data).
However, instead of optimising the sum of errors, it is actually more common to optimise the average error. This means that the update in the online setting remains the same, whereas the update for the full batch is reduced by a factor $N$:
$$\frac{1}{N} \sum_{i=1}^N \nabla_\theta L(g(\boldsymbol{x}_i \mathbin{;} \theta), \boldsymbol{y}_i).$$
In this case, the update in the full batch setting is much smaller (in the same scale as a single update in the online setting, if you want). This also means that more updates leads to better performance.
The figure below illustrates the learning curves when trained on the average error. Note that the performance depends on the number of updates (and not on the number of passes through the data).
It is difficult to prove anything and many results assume some network architectures, unless you can use fit everything into 1 batch.
But from my experience, for small batch sizes relative to the dataset, you want smaller learning rate than when using larger batch sizes (intuitively, you want the general direction to be the sum of many small vectors). Problem is nobody knows what is small enough or large enough is, but we have general idea for some network architectures.
Furthermore, with many training algorithms, the learning rates can adapt and it can get really complex to prove anything.
To improve your training speed, do make batch size as large as possible unless you can fit your entire data into one batch, then breaking it down to mini batches may help you break out of the first local minimum you encounter. There is nothing, however, that guarantees that the other minimums are better than your first local minimum. It is an empirical result really...
