This question already has an answer here:

I am training a deep neural networks for self driving cars using Adam optimization, and I wonder how can I find a standard batch size value , currently I am using the value 1 and I can see that my resources are not fully used (CPU and RAM) ? Thank you


marked as duplicate by kjetil b halvorsen, Peter Flom May 8 '18 at 11:17

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    $\begingroup$ I'm not sure this is more thoroughly-developed than guess & check $\endgroup$ – Sycorax May 7 '18 at 23:20
  • $\begingroup$ 1 is a bit small. usually somewhere between 8 and 128 is reasonable. $\endgroup$ – shimao May 7 '18 at 23:59
  • $\begingroup$ Value 1 will give you very erratic gradients. Might take too long time to converge (if ever) to a useful local maximum. Too large batch might be taxing computationally (and memory-wise on a GPU) and also lead to overfitting. I believe there were works showing that SGD with a good batch size pick acts as a regularizer itself! $\endgroup$ – Vladislavs Dovgalecs May 8 '18 at 0:08
  • $\begingroup$ Probably has an answer already in this thread: stats.stackexchange.com/questions/164876/…. Also, training deep nets on a CPU is not the efficient way of doing things. $\endgroup$ – Jan Kukacka May 8 '18 at 6:32
  • $\begingroup$ JanKukacka I am not training on CPU I ment is there any way to exploit all the CPU during GPU training , $\endgroup$ – ob21 May 8 '18 at 13:59