I have been told to randomize my training set.

Why is it not a great idea to train a classifier by giving it first all the example of class1, then class2, class3 and so on?

Why is it not a great idea to train a classifier by giving it first all the example of class1, then class2, class3 and so on?

I assume you use mini-batch. The error computed after the forward pass on a mini-batch by comparing the predicted outputs with the target outputs should ideally be as close as possible to the error computed on the entire training set, since that is what the backpropagation algorithm minimizes or maximizes.

Quote from {1}:

As for any stochastic gradient descent method (including the mini-batch case), it is important for efficiency of the estimator that each example or minibatch be sampled approximately independently. Because random access to memory (or even worse, to disk) is expensive, a good approximation, called incremental gradient (Bertsekas, 2010), is to visit the examples (or mini-batches) in a fixed order corresponding to their order in memory or disk (repeating the examples in the same order on a second epoch, if we are not in the pure online case where each example is visited only once). In this context, it is safer if the examples or mini-batches are first put in a random order (to make sure this is the case, it could be useful to first shuffle the examples). Faster convergence has been observed if the order in which the mini-batches are visited is changed for each epoch, which can be reasonably efficient if the training set holds in computer memory.

{1} Bengio, Yoshua. "Practical recommendations for gradient-based training of deep architectures." Neural Networks: Tricks of the Trade. Springer Berlin Heidelberg, 2012. 437-478.

To complement the existing answers, one way to think about it is that it protects from "worst case scenario" behaviors.

If we are thinking of iteratively trying to get closer to a minimum following a path based on successive updates, then we want representatives from all classes to pull in their preferred directions. Letting a single class dictate the direction of the path for too long might send the path "to far" and in a worst case scenario to a local optimum from which the optimization problem would not be able to recover even after seeing the other classes' observations.

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.