What impact does mini-batch size have on Adam Optimizer? Is there a recommended mini-batch size when training a (covolutional) neural network with Adam Optimizer?

From what I understood (I might be wrong), for small mini-batch sizes the results tend to be noisy, but the results are also innacurrate for big batches (let's say the whole training set in one pass).


Yes, batch size affects Adam optimizer. Common batch sizes 16, 32, and 64 can be used. Results show that there is a sweet spot for batch size, where a model performs best. For example, on MNIST data, three different batch sizes gave different accuracy as shown in the table below:

|Batch Size | Test Accuracy        |
|1024       |  96% with 30 epochs  |
|64         |  98% with 30 epochs  |
|2          |  99% with 30 epochs  |

Therefore, it can be concluded that decreasing batch size increases test accuracy. However, do not generalize these findings, as it depends on the complexity of on hand data.

Here is a detailed blog (Effect of batch size on training dynamics) that discusses impact of batch size. In addition, following research paper throw detailed overview and analysis how batch size impacts model accuracy (generalization).

  1. Smith, Samuel L., et al. "Don't decay the learning rate, increase the batch size." arXiv preprint arXiv:1711.00489 (2017).
  2. Hoffer, Elad, Itay Hubara, and Daniel Soudry. "Train longer, generalize better: closing the generalization gap in large batch training of neural networks." Advances in Neural Information Processing Systems. 2017.
  • $\begingroup$ +1 for a nice answer! Thanks for leaving the references, I would sure like to read those. Adding to this, decreasing batch size surely slow down model training, right? $\endgroup$ – kamilazdybal Oct 23 '19 at 9:37
  • 1
    $\begingroup$ Yes, compute time increases upon decreasing the batch size. $\endgroup$ – DataFramed Oct 23 '19 at 20:47

I would just leave this as a comment, but I don't have enough reputation.

There's an excellent discussion of the trade offs of large and small batch sizes here.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.