I am training a deep neural networks for self driving cars using Adam optimization, and I wonder how can I find a standard batch size value , currently I am using the value 1 and I can see that my resources are not fully used (CPU and RAM) ? Thank you

  • 1
    $\begingroup$ I'm not sure this is more thoroughly-developed than guess & check $\endgroup$
    – Sycorax
    Commented May 7, 2018 at 23:20
  • $\begingroup$ 1 is a bit small. usually somewhere between 8 and 128 is reasonable. $\endgroup$
    – shimao
    Commented May 7, 2018 at 23:59
  • $\begingroup$ Value 1 will give you very erratic gradients. Might take too long time to converge (if ever) to a useful local maximum. Too large batch might be taxing computationally (and memory-wise on a GPU) and also lead to overfitting. I believe there were works showing that SGD with a good batch size pick acts as a regularizer itself! $\endgroup$ Commented May 8, 2018 at 0:08
  • $\begingroup$ Probably has an answer already in this thread: stats.stackexchange.com/questions/164876/…. Also, training deep nets on a CPU is not the efficient way of doing things. $\endgroup$ Commented May 8, 2018 at 6:32
  • $\begingroup$ JanKukacka I am not training on CPU I ment is there any way to exploit all the CPU during GPU training , $\endgroup$
    – ob21
    Commented May 8, 2018 at 13:59