When the data set size is not a multiple of the mini-batch size, should the last mini-batch be smaller, or contain samples from other batches?

When training a artificial neural network using stochastic gradient descent with mini-batches, if the data set size is not a multiple of mini-batches, should the last mini-batch contains fewer samples? Or instead is it preferable to have the last mini-batch contain the same number of samples as the other batches, by randomly adding samples from other batches (which is the strategy used here and here)?

• Sure, but those double-sampled samples were selected randomly. Yes on that iteration the double-sampled ones will get more weight, but all samples have the same chance of being double-sampled on a given epoch. If N is the total number of samples, and N_missing = minibatch_size - N mod minibatch_size, then each sample will be on average selected (N+N_missing)/N times in a given epoch. – Peter May 13 '18 at 15:24