2
$\begingroup$

I am a newbie in Deep Learning libraries and thus decided to go with Keras. While implementing a NN model, I saw the batch_size parameter in model.fit().

Now, I was wondering if I use the SGD optimizer, and then set the batch_size = 1, m and b, where m = no. of training examples and 1 < b < m, then I would be actually implementing Stochastic, Batch and Mini-Batch Gradient Descent respectively. However, on the other hand, I felt using SGD as the optimizer would by default ignore the batch_size parameter, since SGD stands for Stochastic Gradient Descent and it should always use a batch_size of 1 (i.e use a single data point for each iteration of gradient descent).

I would be grateful if someone could clarify as to which of the above two cases is true.

$\endgroup$
2
$\begingroup$

It works just as you suggest. batch_size parameter does exactly what you would expect: it sets the size of the batch:

  • batch_size: Integer or None. Number of samples per gradient update. If unspecified, batch_size will default to 32.

From programming point of view, Keras decouples the weight update formula parameters specific to each optimizer (learning rate, momentum, etc.) from the global training properties (batch size, training length, etc.) that are share between methods. It is matter of convenience—there is no point in having optimizers SGD, MBGD, BGD that all do the same thing just with different batch size.

$\endgroup$
  • $\begingroup$ Thanks for the reply. This cleared things up. $\endgroup$ – Rajdeep Dutta May 2 at 9:25
  • $\begingroup$ Sorry, but how does this specify Gradient vs Stochastic Gradient versus mini-batch Gradient? A setting of 1 would be regular Gradient descent, right? A setting other than 1 would be Stochastic. There is no way to use mini-batch, as you are not saying "never use all samples". You are only saying "update after this many samples" $\endgroup$ – VISQL Oct 8 at 15:27
  • 1
    $\begingroup$ If you specify batch_size to be the size of the whole dataset, you get batch gradient descent (i.e. nothing stochastic). Anything smaller is mini-batch g.d., which is stochastic. Whether we call the special case of batch size = 1 something else, that's just a matter of nomenclature. $\endgroup$ – Jan Kukacka Oct 8 at 16:04
  • $\begingroup$ @JanKukacka, thank you for clearing this up. I had to re-read a few things as well. $\endgroup$ – VISQL Oct 9 at 9:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.