As is known to all, SVM can use kernel method to project data points in higher spaces so that points can be separated by a linear space. But we can also use logistic regression to choose this boundary in the kernel space, so what's the advantages of SVM? Since SVM uses a sparse model in which only those support vectors make contributions when predicting, does this make SVM faster in prediction?
KLRs and SVMs
- Classification performance is almost identical in both cases.
- KLR can provide class probabilities whereas SVM is a deterministic classifier.
- KLR has a natural extension to multi-class classification whereas in SVM, there are multiple ways to extend it to multi-class classification (and it is still an area of research whether there is a version which has provably superior qualities over the others).
- Surprisingly or unsurprisingly, KLR also has optimal margin properties that the SVMs enjoy (well in the limit at least)!
Looking at the above it almost feels like kernel logistic regression is what you should be using. However, there are certain advantages that SVMs enjoy
- KLR is computationally more expensive than SVM - $O(N^3)$ vs $O(N^2k)$ where $k$ is the number of support vectors.
- The classifier in SVM is designed such that it is defined only in terms of the support vectors, whereas in KLR, the classifier is defined over all the points and not just the support vectors. This allows SVMs to enjoy some natural speed-ups (in terms of efficient code-writing) that is hard to achieve for KLR.
Here's my take on the issue:
SVMs are a very elegant way to do classification. There's some nice theory, some beautiful math, they generalize well, and they're not too slow either. Try to use them for regression though, and it gets messy.
- Here's a resource on SVM regression. Notice the extra parameters to twiddle and the in depth discussion about optimization algorithms.
Gaussian Process Regression has a lot of the same kernelly math, and it works great for regression. Again, the very elegant, and it's not too slow. Try to use them for classification, and it starts feeling pretty kludgy.
Here's a chapter from the GP book on regression.
Here's a chapter on classification, for comparison. Notice that you end up with some complicated approximations or an iterative method.
One nice thing about using GPs for classification, though, is that it gives you a predictive distribution, rather than a simple yes/no classification.
please visit http://www.stanford.edu/~hastie/Papers/svmtalk.pdf
Some conclusions: The classiﬁcation performance is very similar. Has limiting optimal margin properties. Provides estimates of the class probabilities. Often these are more useful than the classiﬁcations. Generalizes naturally to M-class classiﬁcation through kernel multi-logit regression.