4
$\begingroup$

Batch size and the number of iterations are considered as a tradeoff.

It has been observed in practice that when using a larger batch there is a significant degradation in the quality of the model, as measured by its ability to generalize.

(...)

In contrast, small-batch methods consistently converge to flat minimizers, and our experiments support a commonly held view that this is due to the inherent noise in the gradient estimation.

source, found on this thread answer by Frank Dernoncourt.

What if, like when we reduce the learning rate when approaching local minima, we increase the batch size over the number of epochs ? Or increasing it when the loss isn't moving significantly (Netsterov like) ?

$\endgroup$
1

1 Answer 1

4
$\begingroup$

It turns out that increasing batch size during training (in every or alternate epoch) keeping learning rate constant works exactly the same as if batch size was constant and learning rate was decreasing. You can clearly see that in the image below taken from Samuel L. Smith et al. The only difference you'd see is the decrease in the number of parameter updates.

You can get the paper, Don't Decay the Learning Rate, Increase the Batch Size, here.

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.