3
$\begingroup$

SVM is actually a single layer neural network, with identity activation and squared regularized hinge loss, and can be optimized with gradients. In addition, squared regularized hinge loss can be transformed into dual form to induce kernel and find the support vector.

Compared with softmax+cross entropy, squared regularized hinge loss has better convergence and better sparsity.

Why softmax+cross entropy is more dominant in neural network? Why not use squared regularized hinge loss for the CNN?

$\endgroup$
1
  • 2
    $\begingroup$ One reason is that softmax gives an interpretable probability distribution over the different classes. That can be nice to have. $\endgroup$
    – Aaron
    Aug 26 '17 at 19:49
0
$\begingroup$

This guy does an excellent job of working through the math and explanations from intuition and first principles. Take a peek.

tl;dr
Hinge stops penalizing errors after the result is "good enough," while cross entropy will penalize as long as the label and predicted distributions are not identical. The choice of cross-entropy entails that we aiming at the asymptote of perfection, not the threshold of "good enough."

$\endgroup$
1
  • 4
    $\begingroup$ please add a reference for your link if possible $\endgroup$
    – Antoine
    Jan 8 at 18:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.