Whenever we have a multiclass prediction the classifier generates a vector output. Per the definition of a Jacobian we are actually taking Jacobian steps towards a local minimum - so should it technically be called Jacobian Descent?



Although it is a multi-class classification problem, the loss is still a single number, not a vector.

We use cross entropy to convert a vector output to a single number.

This is a very good tutorial I suggest to read.

Here is an example from this tutorial: suppose we have 3 classes and 4 data points. The model output is

enter image description here

And the ground truth is

enter image description here

Cross entropy is

enter image description here

And final loss is sum of this length 4 cross entropy vector.

| cite | improve this answer | |
  • $\begingroup$ Thanks, one quick clarification.. suppose we a multi-layer neural net, is it fair to say that the middle layers have vector loss? and therefore we are performing Jacobian updates? $\endgroup$ – A.D Nov 27 '17 at 17:05
  • $\begingroup$ No. All the numbers in the middle layer are "weights", back prop is just a way to calculate the gradient respect to one single loss. @A.D $\endgroup$ – Haitao Du Nov 27 '17 at 17:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.