When training a artificial neural network using stochastic gradient descent with mini-batches, if the data set size is not a multiple of mini-batches, should the last mini-batch contains fewer samples? Or instead is it preferable to have the last mini-batch contain the same number of samples as the other batches, by randomly adding samples from other batches (which is the strategy used here and here)?


Same number, otherwise you're putting more weight on the samples in the final minibatch (unless you scale down the learning weight to match the smaller size).

Adding random samples from the training set should be fine too (as long as your sampling pool includes the runt minibatch), since each sample has an equal chance of being seen twice in an epoch.

Or just do a modulo and grab samples from the beginning again.

In practice, it probably doesn't matter much.

| cite | improve this answer | |
  • $\begingroup$ Is this true? If the number is different, more weight will be put on each of the indivudual samples in the final minibatch; but that will happen with any repeated samples if you pad the minibatch with already used ones. No? $\endgroup$ – orome Apr 8 '16 at 20:30
  • $\begingroup$ Sure, but those double-sampled samples were selected randomly. Yes on that iteration the double-sampled ones will get more weight, but all samples have the same chance of being double-sampled on a given epoch. If N is the total number of samples, and N_missing = minibatch_size - N mod minibatch_size, then each sample will be on average selected (N+N_missing)/N times in a given epoch. $\endgroup$ – Peter May 13 '18 at 15:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.