4
$\begingroup$

I saw some related question regarding to the fact is one should use sampling with resampling or without when using minibatch.

However my question is different.

Let's assume that I use sampling witouth replacement. Meaning that for each epoch one training example can only belong one 'batch size' group.

I want to know if one should only resample the training data (into the different batch size) only 1 time or before to start to update the parameters before each epoch.

For instance suppose I have 20 'batch size' group. Suppose at the first iteration I have sampled picture A in group 2 and picture B in group 4.

If so should picture A always stay in group 2 and picture B in group 4. Or should I resample it before to start a new epoch (e.g. at epoch 2 put picture B in group 6 and picture B in group 20).

$\endgroup$
  • $\begingroup$ I don't understand what you're asking. Are you asking whether mini-batches are sampled with or without replacement? Or are you asking something else? $\endgroup$ – Sycorax Dec 19 '18 at 16:17
  • $\begingroup$ @Sycorax I edited the post and added an extra assumption. Hope it helps $\endgroup$ – S12000 Dec 19 '18 at 16:23
5
$\begingroup$

The typical usage of SGD is that each minibatch is constructed completely at random without replacement. (There are other ways to construct minibatches; for some comparison of alternatives, see Why do neural network researchers care about epochs?)

Suppose that you have 6 samples: $S=\{A, B, C, D, E, F\}$ and minibatch size 2. The first minibatch could be $\{A, C\}$ and the second $\{D, F\}$ and the third $\{B, E\}$. So what's happening is that at the first minibatch is that you're sampling 2 examples without replacement from the set $S$. At the second minibatch, you're sampling 2 examples without replacement from $S\setminus\{A,C\}=\{B,D,E,F\}$. At the third minibatch, you're sampling 2 examples without replacement from $S\setminus\{A,C,D,F\}=\{B,E\}$. At this final step, there is only a single possible minibatch because you have a minibatch size of 2, you're sampling without replacement, and only 2 examples remain.

Now you've exhausted all of your training samples, so the next epoch starts. Epochs are constructed completely at random, so a valid sequence of minibatches is first $\{D, F\}$, second $\{A, B\}$ and third $\{C,E\}$. It's ok that one of the minibatches with $\{D,F\}$ appear in both epoch 1 and epoch 2 -- this happened purely due to randomness. (And in a more realistic usage where more training data is available, the less likely it is that consecutive epochs will contain one or more mini-batches that are identical, unless the mini-batch size is 1.)

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks, just one question you say 'and batch size 2' but then you give first second and third mini-batch. Thus, I think it is an erratum you have to replace size 2 by size 3 $\endgroup$ – S12000 Dec 19 '18 at 16:48
  • $\begingroup$ No, the batch size refers to how many samples area in each batch. All of the minibatches have 2 samples in them, so the minibatch size is 2. $\endgroup$ – Sycorax Dec 19 '18 at 16:49
  • $\begingroup$ @Syncorax Ahhhhhhh. I got it. Thanks so much for your answer. It was very very helpful. Thanks. $\endgroup$ – S12000 Dec 19 '18 at 16:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.