How does batch size affect Adam Optimizer?

Question

What impact does mini-batch size have on Adam Optimizer? Is there a recommended mini-batch size when training a (covolutional) neural network with Adam Optimizer?

From what I understood (I might be wrong), for small mini-batch sizes the results tend to be noisy, but the results are also innacurrate for big batches (let's say the whole training set in one pass).

Answer 1

Yes, batch size affects Adam optimizer. Common batch sizes 16, 32, and 64 can be used. Results show that there is a sweet spot for batch size, where a model performs best. For example, on MNIST data, three different batch sizes gave different accuracy as shown in the table below:

|Batch Size | Test Accuracy        |
-----------------------------------|
|1024       |  96% with 30 epochs  |
|64         |  98% with 30 epochs  |
|2          |  99% with 30 epochs  |
|__________________________________|

Therefore, it can be concluded that decreasing batch size increases test accuracy. However, do not generalize these findings, as it depends on the complexity of on hand data.

Here is a detailed blog (Effect of batch size on training dynamics) that discusses impact of batch size. In addition, following research paper throw detailed overview and analysis how batch size impacts model accuracy (generalization).

1. Smith, Samuel L., et al. "Don't decay the learning rate, increase the batch size." arXiv preprint arXiv:1711.00489 (2017).
2. Hoffer, Elad, Itay Hubara, and Daniel Soudry. "Train longer, generalize better: closing the generalization gap in large batch training of neural networks." Advances in Neural Information Processing Systems. 2017.

Answer 2

I would just leave this as a comment, but I don't have enough reputation.

There's an excellent discussion of the trade offs of large and small batch sizes here.