# Neural networks: why do we randomize the training set?

I have been told to randomize my training set.

Why is it not a great idea to train a classifier by giving it first all the example of class1, then class2, class3 and so on?

Why is it not a great idea to train a classifier by giving it first all the example of class1, then class2, class3 and so on?

I assume you use mini-batch. The error computed after the forward pass on a mini-batch by comparing the predicted outputs with the target outputs should ideally be as close as possible to the error computed on the entire training set, since that is what the backpropagation algorithm minimizes or maximizes.

Quote from {1}:

As for any stochastic gradient descent method (including the mini-batch case), it is important for efficiency of the estimator that each example or minibatch be sampled approximately independently. Because random access to memory (or even worse, to disk) is expensive, a good approximation, called incremental gradient (Bertsekas, 2010), is to visit the examples (or mini-batches) in a fixed order corresponding to their order in memory or disk (repeating the examples in the same order on a second epoch, if we are not in the pure online case where each example is visited only once). In this context, it is safer if the examples or mini-batches are first put in a random order (to make sure this is the case, it could be useful to first shuffle the examples). Faster convergence has been observed if the order in which the mini-batches are visited is changed for each epoch, which can be reasonably efficient if the training set holds in computer memory.

{1} Bengio, Yoshua. "Practical recommendations for gradient-based training of deep architectures." Neural Networks: Tricks of the Trade. Springer Berlin Heidelberg, 2012. 437-478.

• The answer at Micro vs weighted F1 score seems a bit like a comment to me, @Frank, but it is sort of an answer. I notice you hadn't accepted it. Do you see it as valuable to that thread? I can convert it to a comment, ask the OP to elaborate, or leave it as is. I'd be interested in your opinion. – gung Mar 27 at 14:22

To complement the existing answers, one way to think about it is that it protects from "worst case scenario" behaviors.

If we are thinking of iteratively trying to get closer to a minimum following a path based on successive updates, then we want representatives from all classes to pull in their preferred directions. Letting a single class dictate the direction of the path for too long might send the path "to far" and in a worst case scenario to a local optimum from which the optimization problem would not be able to recover even after seeing the other classes' observations.