Mixing shuffle order on training set for future epochs

Question

Sometimes I like to train a neural network model on a number of epochs, save it, and load it again at a later date for more training (on the exact same training set). During the prior training I'll have the order of the training set randomly shuffled.

When I later resume the saved model, the shuffled order of the training set is often lost. If I resume training with the same training set, but in a different shuffled order, will this lead to any problems for the future epochs? Again, this is the exact same training set (no data added to it, just shuffled differently for future epochs).

Answer 1

The exact oppositie! It's very effective to shuffle your training data between iterations. This makes sure your neural network is not remembering a specific order. And when you don't shuffle training data, your network will actually view the last trained sets as more important.

It is extremely important to shuffle the training data, so that you do not obtain entire minibatches of highly correlated examples. As long as the data has been shuffled, everything should work OK. Different random orderings will perform slightly differently from each other but this will be a small factor that does not matter much.

@Ian Goodfellow, Lead author of the Deep Learning textbook: http://www.deeplearningbook.org

Depending on the problem your neural network has to solve, I advise you to shuffle your training data every iteration or every X iterations.

From here

Check out this answer as well!