# How does batch size affect Adam Optimizer?

What impact does mini-batch size have on Adam Optimizer? Is there a recommended mini-batch size when training a (covolutional) neural network with Adam Optimizer?

From what I understood (I might be wrong), for small mini-batch sizes the results tend to be noisy, but the results are also innacurrate for big batches (let's say the whole training set in one pass).

Yes, batch size affects Adam optimizer. Common batch sizes 16, 32, and 64 can be used. Results show that there is a sweet spot for batch size, where a model performs best. For example, on MNIST data, three different batch sizes gave different accuracy as shown in the table below:

|Batch Size | Test Accuracy        |
-----------------------------------|
|1024       |  96% with 30 epochs  |
|64         |  98% with 30 epochs  |
|2          |  99% with 30 epochs  |
|__________________________________|


Therefore, it can be concluded that decreasing batch size increases test accuracy. However, do not generalize these findings, as it depends on the complexity of on hand data.

Here is a detailed blog (Effect of batch size on training dynamics) that discusses impact of batch size. In addition, following research paper throw detailed overview and analysis how batch size impacts model accuracy (generalization).

• +1 for a nice answer! Thanks for leaving the references, I would sure like to read those. Adding to this, decreasing batch size surely slow down model training, right? – kamilazdybal Oct 23 '19 at 9:37
• Yes, compute time increases upon decreasing the batch size. – DataFramed Oct 23 '19 at 20:47

I would just leave this as a comment, but I don't have enough reputation.

There's an excellent discussion of the trade offs of large and small batch sizes here.