How are samples selected when batching?

Question

Suppose after being randomized samples in a dataset can be written $\{x_i\}_{i = 0}^n$ and are put into a batch for mini-batch gradient descent. Are the samples drawn randomly into a batch, or are the batches like a sliding window of size batch size, such that if the batch size is 32, then for the first batch samples $\{x_i\}_{i = 0}^{31}$ are used, we take a step towards the loss function's global minimum, then for the second samples $\{x_i\}_{i = 32}^{64}$ samples are used, another step, and so on. For the last batch, samples $\{x_i\}_{i = n-32}^{n}$ are used. Once $n$ samples are used, the dataset has ran through a full epoch, the samples are randomized again, and the process is repeated.

Is this the correct way of thinking about how the mini-batch GD process works, or is it gone about a different way, like randomly sampling 32 samples each time?

What happens differently when replacement is allowed?

Answer 1

Yes, one way to think about it is what you described. Another way is: the samples are randomly sorted, then the first 32 are taken, next 32 and so on. Once the epoch is complete, they are randomly sorted and the first 32 are selected, and basically the process repeats.

In some variations replacement is allowed, but most of the time it is not, as far as I know.