In stochastic gradient descent, while decreasing the learning rate during training (e.g., after a set number of epochs) seems to be a common practice, but I haven't seen anyone alter the minibatch size in the same manner. Then I found a paper, On the importance of initialization and momentum in deep learning (Sutskever et al 2013), which got me wondering about it.
In practice, the “transient phase” of convergence (Darken & Moody, 1993), which occurs before fine local convergence sets in, seems to matter a lot more for optimizing deep neural networks. In this transient phase of learning, directions of reduction in the objective tend to persist across many successive gradient estimates and are not completely swamped by noise
So it sounds like there's a post-transient period where noise in the gradient estimate is the dominant feature. A higher minibatch size would decrease this noise. Therefore I started wondering, does it make sense to increase the minibatch size later in the training process? And if so, why doesn't anybody do it?
