I'm given a dataset of 200 million training examples. The stochastic gradient descent method requires me to sample these randomly, to avoid it gets 'stuck'.
First and for all, I don't see how it gets stuck. So the fact the sample needs to be random, and I cannot just traverse the dataset in a sequential manner is a riddle to me right now.
The following paragraph I found here is not clear enough.
The first step of the procedure requires that the order of the training dataset is randomized. This is to mix up the order that updates are made to the coefficients. Because the coefficients are updated after every training instance, the updates will be noisy jumping all over the place, and so will the corresponding cost function. By mixing up the order for the updates to the coefficients, it harnesses this random walk and avoids it getting distracted or stuck.
I would assume entries in a dataset are independent from each other?
But more importantly, if I'm tasked to shuffle this dataset of 200 million examples, does it not introduce a big overhead? Surely shuffling a dataset of 200 million samples is gonna take some time?