Modern neural networks use cross entropy as a loss function, because of shortcomings of MSE. So, why is MSE still being used in SVM (and maybe other learners) and not replaced by cross entropy?

  • $\begingroup$ cross entropy is preferred for classification, while mse is preferred for regression $\endgroup$ – Thomas W Jun 8 '17 at 13:26
  • $\begingroup$ stackoverflow.com/questions/36515202/… $\endgroup$ – Thomas W Jun 8 '17 at 13:27
  • $\begingroup$ @ThomasW You mean it's possible to use CE in SVM? $\endgroup$ – Mehrin Jun 8 '17 at 15:14
  • 1
    $\begingroup$ I'm not familiar with SVM. But I just wanted to point out that Modern neural networks use cross entropy as a loss function is not true. It just depends on the task $\endgroup$ – Thomas W Jun 8 '17 at 15:30

Neural nets are a broad class of models, and many different loss functions can be used. In fact, SVMs can be thought of as a particular kind of shallow neural net. SVMs are much narrower in scope than neural nets. You can't just pop in any arbitrary loss function, or you'd no longer have an SVM. Furthermore, cross entropy is defined on probability distributions, so it can only be used as a loss function when the classifier gives a probability distribution over classes. SVMs don't do this, so cross entropy won't work. SVMs don't use MSE, they use the hinge loss, which gives them their maximum margin properties.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.