When training a artificial neural network using stochastic gradient descent with mini-batches, if the data set size is not a multiple of mini-batches, should the last mini-batch contains fewer samples? Or instead is it preferable to have the last mini-batch contain the same number of samples as the other batches, by randomly adding samples from other batches (which is the strategy used here and here)?
$\begingroup$
$\endgroup$
asked Dec 20, 2015 at 2:47
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
2
Same number, otherwise you're putting more weight on the samples in the final minibatch (unless you scale down the learning weight to match the smaller size).
Adding random samples from the training set should be fine too (as long as your sampling pool includes the runt minibatch), since each sample has an equal chance of being seen twice in an epoch.
Or just do a modulo and grab samples from the beginning again.
In practice, it probably doesn't matter much.
-
$\begingroup$ Is this true? If the number is different, more weight will be put on each of the indivudual samples in the final minibatch; but that will happen with any repeated samples if you pad the minibatch with already used ones. No? $\endgroup$– oromeApr 8, 2016 at 20:30
-
$\begingroup$ Sure, but those double-sampled samples were selected randomly. Yes on that iteration the double-sampled ones will get more weight, but all samples have the same chance of being double-sampled on a given epoch. If
Nis the total number of samples, andN_missing = minibatch_size - N mod minibatch_size, then each sample will be on average selected(N+N_missing)/Ntimes in a given epoch. $\endgroup$– PeterMay 13, 2018 at 15:24