# How are samples selected when batching?

Suppose after being randomized samples in a dataset can be written $$\{x_i\}_{i = 0}^n$$ and are put into a batch for mini-batch gradient descent. Are the samples drawn randomly into a batch, or are the batches like a sliding window of size batch size, such that if the batch size is 32, then for the first batch samples $$\{x_i\}_{i = 0}^{31}$$ are used, we take a step towards the loss function's global minimum, then for the second samples $$\{x_i\}_{i = 32}^{64}$$ samples are used, another step, and so on. For the last batch, samples $$\{x_i\}_{i = n-32}^{n}$$ are used. Once $$n$$ samples are used, the dataset has ran through a full epoch, the samples are randomized again, and the process is repeated.

Is this the correct way of thinking about how the mini-batch GD process works, or is it gone about a different way, like randomly sampling 32 samples each time?

What happens differently when replacement is allowed?

• It isn't different, it is what you described. When replacement is allowed, then the first batch might be $\{x_i\}_{i=0}^{31}$ and the second batch could contain elements from the first batch, for example, $\{x_i\}_{i=27}^{58}$. Commented Mar 21, 2022 at 16:59