I was doing CS231n assignments and found a very interesting implementation of mini-batch gradient descent for SVM image classifier assignment.

It is,

for each epoch:

    sampling some random 'batch_size' number of examples from the training data

    grad = finding gradient over these sampled examples

    changing weights using the calculated gradient 

This is pretty weird comparing to the mini-batch gradient descent which is used in neural networks for instance. It's something like :

for each epoch:

    batches = create batches of training data using whatever the batch_size is

    for each batch in batches:

        grad = calculate gradient over examples in this 'batch' of training data

        changing weights using this above calculated gradient

This one makes much more sense to me as compared to the above one because it uses all the examples for making changes in that epoch, unlike the CS231n's code which uses just some random 'batch_size' number of examples in each epoch.

Can someone explain this to me?

  • $\begingroup$ I am wondering if the two methods would lead to the same result if the random sampling is truly random. Please teach me why you think the first makes much more sense? And could you please tell me which course did you refer to? I think the CS231 I found is not what you mentioned. $\endgroup$ – Lerner Zhang Jun 13 '19 at 14:39
  • $\begingroup$ I am referring to Stanford's CS231n Course. Random sampling is truly random yes. And what I am saying is first one is used in the course, but generally when people talk about mini-batch gradient descent, second one is used. $\endgroup$ – Anant Agarwal Jun 13 '19 at 15:03
  • $\begingroup$ I thought some wiggle room is allowed in applying a certain method. $\endgroup$ – Lerner Zhang Jun 13 '19 at 22:28

I'm assuming that you are referring to the train function for the LinearSVM class in linear_classifier.py.

In this case, the loop is iterating through num_iters (number of steps to take when optimizing), not epoch.

  • $\begingroup$ So num_iters is equal to (number_of_examples / batch_size) ? $\endgroup$ – Anant Agarwal Jun 27 '19 at 3:51
  • $\begingroup$ num_iters should be (total_number_of_examples/batch_size)*epoch. So another way to look at it is that the implementation in linear classifier.py(cs231n) has combined the "for each epoch" loop with "for each batch in batches" loop. $\endgroup$ – Yaofeng Wang Jun 28 '19 at 2:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.