As I'm reading from wikipedia, and this Cross Validated question: Gradient for hinge loss multiclass, the gradient value for a training feature set is somewhat straightforward. However if I'm interpreting this correctly, this only gives a gradient value for the weight of the 'true' class. How do I find the gradients for the other weights? That is, if my total weight vector is [W1, W2, ... Wy... Wk], where y is the class of the training sample then what are the gradient/loss values for every weight that isn't Wy?


1 Answer 1


The answer at Gradient for Hinge Loss Multiclass gives you the Gradient per class.
Since the classifier for the $ j $ - th class is given by the row $ j $ of $ W $ (Which is notated at the answer as $ {W}_{j} $) all you need on each iteration is to update $ {W}_{j} $ according to $ {\nabla}_{{W}_{j}} {L}_{i} $ according to train sample $ {x}_{i} $.

Namely, $ {W}_{j}^{\left( k + 1 \right)} = {W}_{j}^{\left( k \right)} - \eta {\nabla}_{ {W}_{j} } {L}_{i} $, Where $ j $ is the index of the updated row, $ k $ is the iteration counter and $ \eta $ is the Step Size / Learning Rate..


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.