In recent years, Convolutional Neural Networks (CNNs) have become the state-of-the-art for object recognition in computer vision. Typically, a CNN consists of several convolutional layers, followed by two fully-connected layers. An intuition behind this is that the convolutional layers learn a better representation of the input data, and the fully connected layers then learn to classify this representation based into a set of labels.

However, before CNNs started to dominate, Support Vector Machines (SVMs) were the state-of-the-art. So it seems sensible to say that an SVM is still a stronger classifier than a two-layer fully-connected neural network. Therefore, I am wondering why state-of-the-art CNNs tend to use the fully connected layers for classification rather than an SVM? In this way, you would have the best of both worlds: a strong feature representation, and a strong classifier, rather than a strong feature representation but only a weak classifier...

Any ideas?


It can be done; an ICML workshop paper, Deep Learning using Linear Support Vector Machines, Tang (2013), did exactly this and found small but consistent improvements. It's also sometimes done to train CNNs in the typical way, but then take the output of a late layer as "features" and train a separate SVM on that.

Note, though, that they were using linear SVMs, and really, the difference between a linear SVM and logistic regression (which is equivalent to a single binary-output layer with sigmoid activation) is quite small. The additional layer of the net, assuming you have enough data to learn it, actually makes the last two layers stronger than a linear SVM — though of course you could do one fully-connected sigmoid or ReLU layer and then put an SVM layer last.

Also, for multiclass outputs, softmax activations are more natural than multiclass SVMs, and I think the SVM approach of that paper is somewhat more computationally intensive. So people usually just don't think it's worth it.


As far I can see, there are at least couple differences:

  1. CNNs are designed to work with image data, while SVM is a more generic classifier;
  2. CNNs extract features while SVM simply maps its input to some high dimensional space where (hopefully) the differences between the classes can be revealed;
  3. Similar to 2., CNNs are deep architectures while SVMs are shallow;
  4. Learning objectives are different: SVMs look to maximize the margin, while CNNs are not (would love to know more)

This being said, SVMs can work as good as CNNs provided good features are used with a good kernel function.

  • 4
    $\begingroup$ I think you may have misunderstood the question; it's about using an "SVM layer" at the end of the CNN. $\endgroup$ – Dougal Aug 20 '15 at 15:51
  • 4
    $\begingroup$ I understand the difference between a CNN and an SVM, but as @Dougal says, I'm asking more about the final layer of a CNN. Typically, this is a fully-connected neural network, but I'm not sure why SVMs aren't used here given that they tend to be stronger than a two-layer neural network. $\endgroup$ – Karnivaurus Aug 20 '15 at 15:58
  • 6
    $\begingroup$ @Karnivaurus Sorry for misreading your question. The idea is not new. Typically the last layer is thrown away and the output of the last layer is used as features in other classification algorithms. Why it is not done consistently and everywhere? The features of the last layer are typically so discriminative that there is no need of a sophisticated black box as SVM, a simple Logistic Regression does the job. This is my vision of things. $\endgroup$ – Vladislavs Dovgalecs Aug 20 '15 at 16:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.