# How does back propagation work when sending in multiple examples into the network at a time?

I'm trying to understand how training neural networks using batches work. I've referred to posts like this thread this and this but they don't fully answer my question.

When you send in a single example, the errors at the output layer are calculated and used to adjust the weights based on backpropagation. But if you send in multiple examples at once, what happens? Are the errors averaged, such that one round of backpropagation is performed for each batch?

• Averaged or summed, but yes, that’s exactly what happens. – Arya McCarthy May 31 at 22:53
• @AryaMcCarthy In this post someone says "it would be nice to train on the whole dataset", as in use the whole dataset as a batch. But how could that be effective, if that would only result in a single update to the weights? Don't weights need to be updated iteratively for finding their optimal value? – John S May 31 at 22:58
• Training on the whole dataset guarantees convergence to a local optimum; stochastic gradient descent does not. You’re still iteratively updating to find the optimal value in (full-)batch gradient descent. Compute loss for whole dataset, compute gradient, update, repeat. – Arya McCarthy May 31 at 23:11
• @AryaMcCarthy This helps significantly, thank you. So in full batch GD we iteratively update until the gradients don't change (but we pass in the same thing on each iteration, the whole dataset), whereas in SGD we update once for every training example? – John S May 31 at 23:20
• Yes, I’ve collected this information into a proper answer below. – Arya McCarthy May 31 at 23:22

The two extremes of this are true (“batch”) gradient descent, which uses the entire dataset for each update (i.e., minibatch size $$N$$); and a batch size of 1, which uses one training example for each update.