I am training a deep neural networks for self driving cars using Adam optimization, and I wonder how can I find a standard batch size value , currently I am using the value 1 and I can see that my resources are not fully used (CPU and RAM) ? Thank you
$\begingroup$
$\endgroup$
6
-
1$\begingroup$ I'm not sure this is more thoroughly-developed than guess & check $\endgroup$– Sycorax ♦May 7, 2018 at 23:20
-
$\begingroup$ 1 is a bit small. usually somewhere between 8 and 128 is reasonable. $\endgroup$– shimaoMay 7, 2018 at 23:59
-
$\begingroup$ Value 1 will give you very erratic gradients. Might take too long time to converge (if ever) to a useful local maximum. Too large batch might be taxing computationally (and memory-wise on a GPU) and also lead to overfitting. I believe there were works showing that SGD with a good batch size pick acts as a regularizer itself! $\endgroup$– Vladislavs DovgalecsMay 8, 2018 at 0:08
-
$\begingroup$ Probably has an answer already in this thread: stats.stackexchange.com/questions/164876/…. Also, training deep nets on a CPU is not the efficient way of doing things. $\endgroup$– Jan KukackaMay 8, 2018 at 6:32
-
$\begingroup$ JanKukacka I am not training on CPU I ment is there any way to exploit all the CPU during GPU training , $\endgroup$– ob21May 8, 2018 at 13:59
|
Show 1 more comment