2
$\begingroup$

I'm trying to understand how training neural networks using batches work. I've referred to posts like this thread this and this but they don't fully answer my question.

When you send in a single example, the errors at the output layer are calculated and used to adjust the weights based on backpropagation. But if you send in multiple examples at once, what happens? Are the errors averaged, such that one round of backpropagation is performed for each batch?

$\endgroup$
5
  • 1
    $\begingroup$ Averaged or summed, but yes, that’s exactly what happens. $\endgroup$ May 31, 2021 at 22:53
  • $\begingroup$ @AryaMcCarthy In this post someone says "it would be nice to train on the whole dataset", as in use the whole dataset as a batch. But how could that be effective, if that would only result in a single update to the weights? Don't weights need to be updated iteratively for finding their optimal value? $\endgroup$
    – John S
    May 31, 2021 at 22:58
  • 1
    $\begingroup$ Training on the whole dataset guarantees convergence to a local optimum; stochastic gradient descent does not. You’re still iteratively updating to find the optimal value in (full-)batch gradient descent. Compute loss for whole dataset, compute gradient, update, repeat. $\endgroup$ May 31, 2021 at 23:11
  • $\begingroup$ @AryaMcCarthy This helps significantly, thank you. So in full batch GD we iteratively update until the gradients don't change (but we pass in the same thing on each iteration, the whole dataset), whereas in SGD we update once for every training example? $\endgroup$
    – John S
    May 31, 2021 at 23:20
  • $\begingroup$ Yes, I’ve collected this information into a proper answer below. $\endgroup$ May 31, 2021 at 23:22

1 Answer 1

3
$\begingroup$

Extending my comment: yes, that’s what happens. The loss is either summed or averaged over the minibatch. You’d then compute the gradient of this reduced value with respect to the parameters. You perform one update per minibatch.

The two extremes of this are true (“batch”) gradient descent, which uses the entire dataset for each update (i.e., minibatch size $N$); and a batch size of 1, which uses one training example for each update.

The former has some nice convergence guarantees but can be demanding in terms of computer memory. For massive datasets, you also may get significant gains in your objective function without needing the whole dataset.

Stochastic gradient descent is an approximation to batch gradient descent that behaves nicely. Even though an individual step might go in the wrong direction (in the parameter space), the average direction often tends toward a local optimum.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.