Gradient descent in neural networks involves the whole dataset for each weights-update step, and it is well known it would be computationally too long and also could make it converge to a local non-global minimum.
(Pure) stochastic gradient descent does a weights-update for each training example:
As the algorithm sweeps through the training set, it performs the above update for each training example. Several passes can be made over the training set until the algorithm converges. If this is done, the data can be shuffled for each pass to prevent cycles
A compromise between 1. and 2. is to use mini-batches, i.e. each weights-update step is done typically for n training examples (usually between 32 and 1000 in what I read).
I know 2. is better than 1. and I have read some examples showing this.
Question: Would you have a reference showing that 3. is better than 2.? i.e. showing that on some NN examples (e.g. MNIST digit recognition), we have a better accuracy at the end when updating weights on mini-batches rather than updating weights for each training example?
Note: I have already read Batch gradient descent versus stochastic gradient descent, Stochastic gradient descent Vs Mini-batch size 1, and Why mini batch size is better than one single “batch” with all training data? (the latter is a comparison 3. vs 1., whereas here I'd like to compare 3. vs 2.)
