21
$\begingroup$

In the mini-batch training of a neural network, I heard that an important practice is to shuffle the training data before every epoch. Can somebody explain why the shuffling at each epoch helps?

From the google search, I found the following answers:

  • it helps the training converge fast
  • it prevents any bias during the training
  • it prevents the model from learning the order of the training

But, I have the difficulty of understanding why any of those effects is caused by the random shuffling. Can anybody provide an intuitive explanation?

$\endgroup$
14
$\begingroup$

Note: throughout this answer I refer to minimization of training loss and I do not discuss stopping criteria such as validation loss. The choice of stopping criteria does not affect the process/concepts described below.

The process of training a neural network is to find the minimum value of a loss function $ℒ_X(W)$, where $W$ represents a matrix (or several matrices) of weights between neurons and $X$ represents the training dataset. I use a subscript for $X$ to indicate that our minimization of $ℒ$ occurs only over the weights $W$ (that is, we are looking for $W$ such that $ℒ$ is minimized) while $X$ is fixed.

Now, if we assume that we have $P$ elements in $W$ (that is, there are $P$ weights in the network), $ℒ$ is a surface in a $P+1$-dimensional space. To give a visual analogue, imagine that we have only two neuron weights ($P=2$). Then $ℒ$ has an easy geometric interpretation: it is a surface in a 3-dimensional space. This arises from the fact that for any given matrices of weights $W$, the loss function can be evaluated on $X$ and that value becomes the elevation of the surface.

But there is the problem of non-convexity; the surface I described will have numerous local minima, and therefore gradient descent algorithms are susceptible to becoming "stuck" in those minima while a deeper/lower/better solution may lie nearby. This is likely to occur if $X$ is unchanged over all training iterations, because the surface is fixed for a given $X$; all its features are static, including its various minima.

A solution to this is mini-batch training combined with shuffling. By shuffling the rows and training on only a subset of them during a given iteration, $X$ changes with every iteration, and it is actually quite possible that no two iterations over the entire sequence of training iterations and epochs will be performed on the exact same $X$. The effect is that the solver can easily "bounce" out of a local minimum. Imagine that the solver is stuck in a local minimum at iteration $i$ with training mini-batch $X_i$. This local minimum corresponds to $ℒ$ evaluated at a particular value of weights; we'll call it $ℒ_{X_i}(W_i)$. On the next iteration the shape of our loss surface actually changes because we are using $X_{i+1}$, that is, $ℒ_{X_{i+1}}(W_i)$ may take on a very different value from $ℒ_{X_i}(W_i)$ and it is quite possible that it does not correspond to a local minimum! We can now compute a gradient update and continue with training. To be clear: the shape of $ℒ_{X_{i+1}}$ will -- in general -- be different from that of $ℒ_{X_{i}}$. Note that here I am referring to the loss function $ℒ$ evaluated on a training set $X$; it is a complete surface defined over all possible values of $W$, rather than the evaluation of that loss (which is just a scalar) for a specific value of $W$. Note also that if mini-batches are used without shuffling there is still a degree of "diversification" of loss surfaces, but there will be a finite (and relatively small) number of unique error surfaces seen by the solver (specifically, it will see the same exact set of mini-batches -- and therefore loss surfaces -- during each epoch).

One thing I deliberately avoided was a discussion of mini-batch sizes, because there are a million opinions on this and it has significant practical implications (greater parallelization can be achieved with larger batches). However, I believe the following is worth mentioning. Because $ℒ$ is evaluated by computing a value for each row of $X$ (and summing or taking the average; i.e., a commutative operator) for a given set of weight matrices $W$, the arrangement of the rows of $X$ has no effect when using full-batch gradient descent (that is, when each batch is the full $X$, and iterations and epochs are the same thing).

$\endgroup$
6
  • $\begingroup$ Thank you for this helpful explanation. After reading your answer deeply, I have two questions: 1. You mentioned that mini-batch supplies a degree of limited diversification. I don't understand why this is not enough to avoid stucking in local minimum. If a the solver is in local minima of the surface of one batch, it is with high probability not in local minima of the surface of the next batch, thus, it should not stuck.? 2. How does the solver converge in the surface of the loss function while the surface always changes by using different batches? $\endgroup$ – Code Pope Jun 18 '19 at 13:33
  • 1
    $\begingroup$ The diversification benefit is still there without shuffling, but it's not as significant as when shuffling is used because it's constantly seeing the same sequence of loss surfaces, whereas if we use shuffling it probably never sees the same exact loss surface more than once. As for stopping criteria, I've generally written mine such that once the average percent reduction in loss over a specified number of iterations is less than some tolerance, the training stops. $\endgroup$ – Josh Jun 18 '19 at 16:18
  • 2
    $\begingroup$ Maybe I didn't really answer your question about local minima so well. In theory, these loss surfaces should all exhibit some degree of similarity since the response's probability distribution (conditional on the model's predictors) is assumed to be constant over the entire training dataset. So if a minimum is deep enough it should show up across a great many mini-batches therefore the solver is unlikely to bounce out of it. But if the minimum is basically just "noise" then this strategy should work pretty well and allow the model to continue training. $\endgroup$ – Josh Jun 18 '19 at 16:22
  • $\begingroup$ Thanks. Your second response makes it clearly understandable why the NN converges despite the different surfaces. Regarding my first question, is it correct to say that having the same sequence would have just a higher possibility that a "noise" repeats in many of the loss surfaces of the batches than when using shuffling? This is the only explanation I can give why it is still possible to get stuck in local minima when using mini batches without shuffling. $\endgroup$ – Code Pope Jun 18 '19 at 18:07
  • $\begingroup$ @CodePope I think that's right. Also, once the loss has been reduced a lot since the beginning of training, the gradients will be pretty small and it may even be possible that the solver basically gets stuck in a "loop" as it keeps seeing the same sequence of loss surfaces. Please note that this is a speculative explanation based on my rather limited experience, so if you have a serious theoretical interest in this you'd better consult an expert. $\endgroup$ – Josh Jun 19 '19 at 9:52
8
$\begingroup$

To try to give another explanation:

One of the most powerful things about neural networks is that they can be very complex functions, allowing one to learn very complex relationships between your input and output data. These relationships can include things you would never expect, such as the order in which data is fed in per epoch. If the order of data within each epoch is the same, then the model may use this as a way of reducing the training error, which is a sort of overfitting.

With respect to speed: Mini-batch methods rely on stochastic gradient descent (and improvements thereon), which means that they rely on the randomness to find a minimum. Shuffling mini-batches makes the gradients more variable, which can help convergence because it increases the likelihood of hitting a good direction (or at least that is how I understand it).

$\endgroup$
2
  • 2
    $\begingroup$ How can a neural network learn the order in which data is fed in each epoch? $\endgroup$ – Code Pope Jun 18 '19 at 13:43
  • 2
    $\begingroup$ In a word, they can't. If using mini-batch training (i.e., more than one batch per epoch), then a particular order to the data may influence training in the sense that by training on one mini-batch first the solver may enter a certain region (perhaps containing a local minimum...) rather than another. But to say that a feedforward NN "learns" about the ordering of data isn't really correct because each prediction is made independently of every other prediction, and the ordering within mini-batches will, of course, have no effect whatsoever. $\endgroup$ – Josh Oct 23 '19 at 15:20
6
$\begingroup$

Imagine your last few minibatch labels indeed have more noise. Then these batches will pull the final learned weights in the wrong direction. If you shuffle every time, the chances of last few batches being disproportionately noisy goes down.

$\endgroup$
6
$\begingroup$

From a very simplistic point of view, the data is fed in sequentially, which suggests that at the very least, it's possible for the data order to have an effect on the output. If the order doesn't matter, randomization certainly won't hurt. If the order does matter, randomization will help to smooth out those random effects so that they don't become systematic bias. In short, randomization is cheap and never hurts, and will often minimize data-ordering effects.

$\endgroup$
1
$\begingroup$

When you train your network using a fixed data set, meaning data you never shuffling during the training. You are very much likely to get weights that are very high and very low such as 40,70,-101,200...etc. This simply means that your network has not learnt the training data but it has learnt the noise of your training data. Classic case of an overfit model. With such network you'll get spot on predictions for the data you have used for training. If you use any other inputs to test it, your model will fall apart. Now, when you shuffle training data after each epoch (iteration of overall set) ,you simply feed different input to neurons at each epoch and that simply regulates the weights meaning you're more likely to get "lower" weights that are closer to zero, and that means your network can make better generalisations.

I hope that was clear.

$\endgroup$
1
$\begingroup$

Here is a more intuitive explanation:

When using gradient descent, we want the loss get reduced in a direction of gradient. The gradient is calculated by the data from a single mini-batch for each round of weight updating. The thing we want happen is this mini-batch-based gradient is roughly the population gradient, because this is expected to produce a quicker convergence. (Imagine if you feed the network 100 class1 data in one mini-batch, and 100 class2 data in another, the network will hover around. A better way is to feed it with 50 class1 + 50 class2 in each mini-batch.)

How to achieve this since we cannot use the population data in a mini-batch? The art of statistics tells us: shuffle the population, and the first batch_size pieces of data can represent the population. This is why we need to shuffle the population.

I have to say, shuffling is not necessary if you have other method to sample data from population and ensure the samples can produce a reasonable gradient.

That's my understanding. Hope it helps.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.