CS231n SVM Optimization : Mini Batch Gradient Descent

Question

I was doing CS231n assignments and found a very interesting implementation of mini-batch gradient descent for SVM image classifier assignment.

It is,

for each epoch:

    sampling some random 'batch_size' number of examples from the training data

    grad = finding gradient over these sampled examples

    changing weights using the calculated gradient 

This is pretty weird comparing to the mini-batch gradient descent which is used in neural networks for instance. It's something like :

for each epoch:

    batches = create batches of training data using whatever the batch_size is

    for each batch in batches:

        grad = calculate gradient over examples in this 'batch' of training data

        changing weights using this above calculated gradient

This one makes much more sense to me as compared to the above one because it uses all the examples for making changes in that epoch, unlike the CS231n's code which uses just some random 'batch_size' number of examples in each epoch.

Can someone explain this to me?

Answer 1

I'm assuming that you are referring to the train function for the LinearSVM class in linear_classifier.py.

In this case, the loop is iterating through num_iters (number of steps to take when optimizing), not epoch.