Sum or average of gradients in (mini) batch gradient decent? [duplicate]

Question

When I implemented mini batch gradient decent, I just averaged the gradients of all examples in the training batch. However, I noticed that now the optimal learning rate is much higher than for online gradient decent. My intuition is that this is because the averaged gradient is less noisy and could thus be followed faster. So maybe it also makes sense just to sum up the gradients of a batch. The values can be positive and negative anyway.

I know it's just a constant factor that can be balanced using the learning rate. But I wonder which is the definition scientists have agreed on so that I can reproduce results from neural network papers.

Does one typically divide the summed gradients of a batch by the batch size?

Answer 1

Average.

Examples: Notes to Andrew Ng's Machine Learning Course on Coursera compiled by Alex Holehouse.

Summing the gradients due to individual samples you get a much smoother gradient. The larger the batch the smoother the resulting gradient used in updating the weight.

Dividing the sum by the batch size and taking the average gradient has the effect of:

1. The magnitude of the weight does not grow out of proportion. Adding L2 regularization to the weight update penalizes large weight values. This often leads to improved generalization performance. Taking the average, especially if the gradients happen to point in the same direction, keep the weights from getting too large.
2. The magnitude of the gradient is independent of the batch size. This allows comparison of weights from other experiments using different batch sizes.
3. Countering the effect of the batch size with the learning rate can be numerically equivalent but you end up with a learning rate that is implementation specific. It makes it difficult to communicate your results and experimental setup if people cannot relate to the scale of parameters you're using and they'll have trouble reproducing your experiment.

Averaging enables clearer comparability and keeping gradient magnitudes independent of batch size. Choosing a batch size is sometimes constrained by the computational resources you have and you want to mitigate the effect of this when evaluating your model.