I was taking Stanford's cs231n class and was unable to understand the gradient calculated using the SVM loss function.

You should go here to check the notes which I am talking about.

This is the SVM loss function. SVM LOSS FUNCTION

And now we want to find it's gradients. With respect to the weights of the correct class, the gradient of the loss function will be, GRADIENT WITH RESPECT TO CORRECT CLASSES Until Now I understand Everything, the problem occurs in finding the gradient with respect to the incorrect classes :


I do not understand the calculation of this gradient, where did the Sigma go?


When you take the derivative wrt some $k\neq y_i$, the $w_k$ appears in the whole expression just once, i.e. when $j=k$. And, the gradient will be just $x_i$ times the indicator.

For example, let the set $j\neq y_i$ be $\{a,b,k\}$. The expanded version of the loss function will be $$L_i=\max(0,w_ax_i-w_{y_i}x_i+\Delta)+\max(0,w_bx_i-w_{y_i}x_i+\Delta)+\max(0,w_kx_i-w_{y_i}x_i+\Delta)$$

If you take the derivative with respect to $w_k$, you'll have just one indicator function, leaving you with no sigma expression.

  • $\begingroup$ Okay so are you saying that when we are talking about 'derivative of L(sub)i w.r.t a particular class j', only one term remains because other classes with 'w(sub)j' will be denoting different classes rather than that same 'j th' class whose respect to we are differentiating. I know I have phrased this poorly but I hope you get me, o/w I'll try and rephrase this some other way, Thanks $\endgroup$ – Anant Agarwal Jun 8 at 12:33
  • $\begingroup$ Yes, exactly, you put it right. $\endgroup$ – gunes Jun 8 at 12:38
  • $\begingroup$ Thanks a ton man. $\endgroup$ – Anant Agarwal Jun 8 at 12:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.