Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I'm trying to understand how training neural networks using batches work. I've referred to posts like this thread this and this but they don't fully answer my question.
When you send in a single example, the errors at the output layer are calculated and used to adjust the weights based on backpropagation. But if you send in multiple examples at once, what happens? Are the errors averaged, such that one round of backpropagation is performed for each batch?
Extending my comment: yes, that’s what happens. The loss is either summed or averaged over the minibatch. You’d then compute the gradient of this reduced value with respect to the parameters. You perform one update per minibatch.
The two extremes of this are true (“batch”) gradient descent, which uses the entire dataset for each update (i.e., minibatch size $N$); and a batch size of 1, which uses one training example for each update.
The former has some nice convergence guarantees but can be demanding in terms of computer memory. For massive datasets, you also may get significant gains in your objective function without needing the whole dataset.
Stochastic gradient descent is an approximation to batch gradient descent that behaves nicely. Even though an individual step might go in the wrong direction (in the parameter space), the average direction often tends toward a local optimum.
