Neural networks: why do we randomize the training set? I have been told to randomize my training set.
Why is it not a great idea to train a classifier by giving it first all the example of class1, then class2, class3 and so on?
 A: 
Why is it not a great idea to train a classifier by giving it first all the example of class1, then class2, class3 and so on?

I assume you use mini-batch. The error computed after the forward pass on a mini-batch by comparing the predicted outputs with the target outputs  should ideally be as close as possible to the error computed on the entire training set, since that is what the backpropagation algorithm minimizes or maximizes.
Quote from {1}:

As for any stochastic gradient descent method (including
  the mini-batch case), it is important for efficiency of the estimator that each example or minibatch
  be sampled approximately independently. Because
  random access to memory (or even worse, to
  disk) is expensive, a good approximation, called incremental
  gradient (Bertsekas, 2010), is to visit the
  examples (or mini-batches) in a fixed order corresponding
  to their order in memory or disk (repeating
  the examples in the same order on a second epoch, if
  we are not in the pure online case where each example
  is visited only once). In this context, it is safer if
  the examples or mini-batches are first put in a random
  order (to make sure this is the case, it could
  be useful to first shuffle the examples). Faster convergence
  has been observed if the order in which the
  mini-batches are visited is changed for each epoch,
  which can be reasonably efficient if the training set
  holds in computer memory.

{1} Bengio, Yoshua. "Practical recommendations for gradient-based training of deep architectures." Neural Networks: Tricks of the Trade. Springer Berlin Heidelberg, 2012. 437-478.
A: To complement the existing answers, one way to think about it is that it protects from "worst case scenario" behaviors.
If we are thinking of iteratively trying to get closer to a minimum following a path based on successive updates, then we want representatives from all classes to pull in their preferred directions. Letting a single class dictate the direction of the path for too long might send the path "to far" and in a worst case scenario to a local optimum from which the optimization problem would not be able to recover even after seeing the other classes' observations.
