1
$\begingroup$

In stochastic gradient descent, while decreasing the learning rate during training (e.g., after a set number of epochs) seems to be a common practice, but I haven't seen anyone alter the minibatch size in the same manner. Then I found a paper, On the importance of initialization and momentum in deep learning (Sutskever et al 2013), which got me wondering about it.

In practice, the “transient phase” of convergence (Darken & Moody, 1993), which occurs before fine local convergence sets in, seems to matter a lot more for optimizing deep neural networks. In this transient phase of learning, directions of reduction in the objective tend to persist across many successive gradient estimates and are not completely swamped by noise

So it sounds like there's a post-transient period where noise in the gradient estimate is the dominant feature. A higher minibatch size would decrease this noise. Therefore I started wondering, does it make sense to increase the minibatch size later in the training process? And if so, why doesn't anybody do it?

|cite|improve this question
$\endgroup$
1
$\begingroup$

SGD noise acts as a regularizer. The smaler the batch size the higher the noise. Without noise you are going to end up in a training loss minimum, but probably get a huge generalization error.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ thanks for posting an answer. I never thought that the error in estimating the objective could be a type of regularization, but it's an interesting idea. But, if I had other means of regularization employed (e.g., dropout), then I could be saved from high generalization error even if I had a perfect loss estimate. It kind of sounds like you're saying "no" to the original question of it making sense to modify the minibatch size during training. Is that so? Can you clarify? $\endgroup$ – Ben Ogorek Jan 11 '18 at 0:28
  • $\begingroup$ Well, for starters you cannot be sure about anything related to nets until it's checked empirically. I believe bigger batch size will have an effect similar to reducing learning rate, but worse performance wise. $\endgroup$ – Lugi Jan 11 '18 at 21:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.