# CS231n SVM Optimization : Mini Batch Gradient Descent

I was doing CS231n assignments and found a very interesting implementation of mini-batch gradient descent for SVM image classifier assignment.

It is,

for each epoch:

sampling some random 'batch_size' number of examples from the training data

grad = finding gradient over these sampled examples

changing weights using the calculated gradient


This is pretty weird comparing to the mini-batch gradient descent which is used in neural networks for instance. It's something like :

for each epoch:

batches = create batches of training data using whatever the batch_size is

for each batch in batches:

grad = calculate gradient over examples in this 'batch' of training data

changing weights using this above calculated gradient


This one makes much more sense to me as compared to the above one because it uses all the examples for making changes in that epoch, unlike the CS231n's code which uses just some random 'batch_size' number of examples in each epoch.

Can someone explain this to me?

• I am wondering if the two methods would lead to the same result if the random sampling is truly random. Please teach me why you think the first makes much more sense? And could you please tell me which course did you refer to? I think the CS231 I found is not what you mentioned. – Lerner Zhang Jun 13 '19 at 14:39
• I am referring to Stanford's CS231n Course. Random sampling is truly random yes. And what I am saying is first one is used in the course, but generally when people talk about mini-batch gradient descent, second one is used. – Anant Agarwal Jun 13 '19 at 15:03
• I thought some wiggle room is allowed in applying a certain method. – Lerner Zhang Jun 13 '19 at 22:28