Presently the method used is, you train a CNN completely and final layer features maps are taken and an SVM is used to classify. Is it possible to train an end to end hybrid CNN-SVM network? Since both have a different loss function, it can be complicated, but is there any paper on the same?

  • 1
    $\begingroup$ Sure. It's been tried - googling "svm on top of neural network" gives this paper. Some DL frameworks have this feature out-of-the-box, for example see MXNet. $\endgroup$ Jan 12, 2018 at 10:34
  • $\begingroup$ +1. IMO this paper is misguided? Roughly isn't it just showing that training with hinge loss gives better test accuracy than cross entropy. [which is to be expected since hinge loss is closer to accuracy metric]? $\endgroup$
    – seanv507
    Jan 1, 2019 at 12:33
  • $\begingroup$ I don't think that this is true in general, as I tried it several times and it performed worse - being close to test metric is not everything, you also need to take into account the problems with gradient descent- especially since hinge loss is not differentiable $\endgroup$ Jan 2, 2019 at 22:08

1 Answer 1


If I understand your question correctly, you're saying that typically after training a CNN with a softmax classifier layer, people then do additional training using an SVM or GBM on the last feature layer, to squeeze out even more accuracy. Typically the network is pretrained using a standard set of neural layers, and then you pass all your images to get the feature layer embedding, after which you train an SVM or GBM from scratch on a now fixed input. However, you would like to train both at the same time.

The answer is yes, it's theoretically possible. The loss function is exactly the same as for your classifier, it's just that you're using an SVM instead of a neural network layer to do the final classification part. However, this can be quite slow. Typical feature layers are on the order of 1000 dimensions. Also, your CNN feature layer changes over time since the network is learning. So that's why people

  • $\begingroup$ I wouldn't postulate the training time of SGD based learning with hinge loss versus cross entropy loss would be too different. Is there anything written on this? $\endgroup$
    – Firebug
    Jan 12, 2018 at 11:20
  • $\begingroup$ @firebug - you should point to the little known fact that svm can be done in primal with sgd (as in previous post of yours) $\endgroup$
    – seanv507
    Jan 1, 2019 at 12:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.