SVM is actually a single layer neural network, with identity activation and squared regularized hinge loss, and can be optimized with gradients. In addition, squared regularized hinge loss can be transformed into dual form to induce kernel and find the support vector.

Compared with softmax+cross entropy, squared regularized hinge loss has better convergence and better sparsity.

Why softmax+cross entropy is more dominant in neural network? Why not use squared regularized hinge loss for the CNN?

  • $\begingroup$ One reason is that softmax gives an interpretable probability distribution over the different classes. That can be nice to have. $\endgroup$ – Aaron Aug 26 '17 at 19:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.