# Does the chronological order of batches and epochs matter?

BACKGROUND

I have a very large file that contains my training data. For the sake of simplicity, assume the data stored in this file is shuffled and stratified so that any idiosyncrasies get as spread-out as possible.

I want to train a neural network on this data file. Typically, a neural network consumes data in batches, then once all the batches are consumed, one starts over from the first batch for the next iteration of training (aka. epoch).

PROBLEM

If the file is so large that only part of it (say, half of it) can be loaded to memory, how should I consume the training data? As far as the training is concerned, what are the pros and cons of the two scenarios below?

Scenario 1:

• For each segment of the file...
• For each epoch...
• For each batch...
• Train the network on the batch.

Scenario 2:

• For each epoch...
• For each segment of the file...
• For each batch...
• Train the network on the batch.

Let's spell these out in more details. Assume that the file is made up of 8 batches b1 b2 b3 b4 b5 b6 b7 b8 and that at most 4 contiguous batches can be loaded to memory. If the training consist of 3 epochs, then scenario 1 would effectively feed the following sequence of batches to the neural network:

First ½ of the file                 Second ½ of the file
|                                   |
epoch 1     epoch 2     epoch 3     epoch 1     epoch 2     epoch 3
|           |           |           |           |           |
b1 b2 b3 b4 b1 b2 b3 b4 b1 b2 b3 b4 b5 b6 b7 b8 b5 b6 b7 b8 b5 b6 b7 b8

On the other hand, scenario 2 would be

epoch 1                 epoch 2                 epoch 3
|                       |                       |
First ½     Second ½    First ½     Second ½    First ½     Second ½
|           |           |           |           |           |
b1 b2 b3 b4 b5 b6 b7 b8 b1 b2 b3 b4 b5 b6 b7 b8 b1 b2 b3 b4 b5 b6 b7 b8

Presumably, scenario 1 first biases the neural network to the batches 1-4 and then adds a bias towards batches 5-8, whereas scenario 2, "epoch-wise" at least, does not introduce any bias since it progresses through all batches 1-8 at each iteration. But the question is, does any of this ordering matter? Is there an altogether better alternative to these two scenarios?

• Stochastic gradient descent (SGD) is a readily available implementation for estimating neural nets without loading the entire dataset into memory. ufldl.stanford.edu/tutorial/supervised/… – AdamO Dec 7 '17 at 13:12
• @AdamO I'm afraid this isn't addressing the problem. SGD does not mean that you don't need to read the whole file, be it in whole or by chunks. What do you do when the file has to be read in chunks because it's too large? Do you train according to scenario 1 or 2? – Tfovid Dec 7 '17 at 13:45

Takeaway message: With SGD and large datasets, epochs are not that important; To avoid keeping entire datasets in memory, just keep generating freshly sampled minibatches and you'll be fine.

With respect to your suggestions, both are not recommended.

• generally, fixed batches are a bad idea.
• your first solution is specifically bad. It is in fact equivalent to training your network on half the data and then fine-tuning on the second half. In some cases this can lead to what is known as "catastrophic forgetting", which is that the finetuning phase will actually make the network "forget" its ability to predict well for the first half of the data.

A better approach

It is a very common practice to train on datasets that are so large they don't fit into memory. Generally what you want to do is implement a generator that prepares "fresh" batches for you. With keras, for example, this is very easy:

1. write a custom generator that yields (examples, labels) with the number of examples being the desired batch size. The generator is expected to loop over your data indefinitely.
2. train your network using the fit_generator method. If you generate a batch of size $b$ and your training set is of size $m$, then a single epoch would consist of $\frac{m}{b}$ steps.

As for the implementation of the generator, which is what your'e asking about, since you're training with SGD you want to sample $b$ random examples every time. Now you have two approaches: sample with replacement and sample without replacement. The naive implementation would just be to sample with replacement (since sampling without replacement would require you to keep track of the previous batches the generator yielded). It's true that in this way you're not guaranteed to go over all the $m$ training examples at every epoch, but in different epochs you'll be "missing" different examples, which will sort-of cancel itself out as training progresses. This tends to be especially insignificant for large datasets (see [2] for a more detailed discussion on this question).

• I am still digesting this whole SGD business. (My whole implementation is with basic TensorFlow using fixed batches so I have to live with that for now.) I'll definitely have to re-factor my code to implement SGD, but in the meanwhile, do you see anything fundamentally wrong with scenario 2? (I kind of understand your criticism about scenario 1, but you didn't say what was wrong with #2. Is it fundamentally a no-go or is it just not as sexy as SGD?) – Tfovid Dec 11 '17 at 8:51
• Actually, going back to scenario 1---it does have the advantage that one can add more training data as it gets acquired. That's because the data file is the "outermost iteration". It's a lot less obvious how new data can be incorporated in scenario 2 since the epoch is then the outermost iteration. If anything, scenario 1 is in that sense better at capturing any shifts of the classification hyperplanes over time. (In which case some "forgetting" of older training data is a good thing.) – Tfovid Dec 11 '17 at 9:25