When we train neural networks, we vectorize all examples and after back propagation , subtract the sum of gradients for all examples per parameter. Would not it be accurate to update parameter for one example, then reuse the calculated parameter to generate updates for the next example. When we calculate all gradients at once and sum them, we may have subtracted more out of the parameters than we would have if we had done step by step. For example if i have say 2 examples, i do -1 and then calculate again and do -.5 . We will have weights as -1.5. But if i did together, I might have done -1 , then for the next example -1.5. For a total of -2.5..
Updating neural networks is often done in batches with a few examples like you say. If you update one example at a time then a lot of the training is used to learn an update for individual examples. By using many examples for each update then the updates are more generalized and the training of weights has less variation.