As is known to all, SVM can use kernel method to project data points in higher spaces so that points can be separated by a linear space. But we can also use logistic regression to choose this boundary in the kernel space, so what's the advantages of SVM? Since SVM uses a sparse model in which only those support vectors make contributions when predicting, does this make SVM faster in prediction?
$\begingroup$
$\endgroup$
1
-
2$\begingroup$ Hastie's slides are what you're looking for $\endgroup$– Yibo YangCommented Mar 9, 2017 at 15:48
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
10
KLRs and SVMs
- Classification performance is almost identical in both cases.
- KLR can provide class probabilities whereas SVM is a deterministic classifier.
- KLR has a natural extension to multi-class classification whereas in SVM, there are multiple ways to extend it to multi-class classification (and it is still an area of research whether there is a version which has provably superior qualities over the others).
- Surprisingly or unsurprisingly, KLR also has optimal margin properties that the SVMs enjoy (well in the limit at least)!
Looking at the above it almost feels like kernel logistic regression is what you should be using. However, there are certain advantages that SVMs enjoy
- KLR is computationally more expensive than SVM - $O(N^3)$ vs $O(N^2k)$ where $k$ is the number of support vectors.
- The classifier in SVM is designed such that it is defined only in terms of the support vectors, whereas in KLR, the classifier is defined over all the points and not just the support vectors. This allows SVMs to enjoy some natural speed-ups (in terms of efficient code-writing) that is hard to achieve for KLR.
-
8$\begingroup$ +1 I would just add though that if computational complexity is an issue, it isn't too difficult to construct a sparse kernel logistic regression model by greedily choosing the basis vectors to minimise the regularised loss on the training set, or other approaches. See the papers on the "Informative Vector Machine" for example. $\endgroup$ Commented Nov 20, 2012 at 9:55
-
5$\begingroup$ Also, quite often if you optimise the kernel and regularisation parameters of an SVM you end up with a model where virtually all of the data are support vectors. The sparsity of SVMs is a happy accident, it isn't really a good selling point of the technique as it is generally possible to achieve greater sparsity by other means. $\endgroup$ Commented Nov 20, 2012 at 9:56
-
1$\begingroup$ @DikranMarsupial Thanks for the pointer to Informative Vector Machine. I know of some works in Sparse KLR but so far I don't think any of them scale well for large datasets. Either way releasing a good implementation of sparse KLR which is user-friendly like libSVM or SVM Light can go a long way in its adoption. Apologies if such implementations already exists, however I am not aware of any.(EDIT: I think you meant "Import vector machine" instead of "Informative vector machine"?) $\endgroup$ Commented Nov 20, 2012 at 10:47
-
2$\begingroup$ If you are ending up with all data points as support vectors, then you are over fitting. This happens with RBF many times. In fact, one of the fundamental thing I have learnt as a user of SVM is to first and foremost check the fraction of points chosen as support vectors. If it is anything more than 30% of the data, I outright reject that model. $\endgroup$ Commented Nov 20, 2012 at 10:48
-
3$\begingroup$ It is not correct that all data points being SVs means over-fitting. If the value of C is small, then there is little penalty on the slack variables then you can have a very bland classifier (that makes many errors on the training set) and the margin is so wide that all the data are support vectors. Rejecting non-sparse models is not a good rule of thumb as sometimes the SVM with the best generalisation performance is non-sparse. The number of SVs is an upper bound on the leave-one-out error, but it is often a very lose bound indeed! $\endgroup$ Commented Nov 20, 2012 at 12:25
$\begingroup$
$\endgroup$
1
Here's my take on the issue:
SVMs are a very elegant way to do classification. There's some nice theory, some beautiful math, they generalize well, and they're not too slow either. Try to use them for regression though, and it gets messy.
- Here's a resource on SVM regression. Notice the extra parameters to twiddle and the in depth discussion about optimization algorithms.
Gaussian Process Regression has a lot of the same kernelly math, and it works great for regression. Again, the very elegant, and it's not too slow. Try to use them for classification, and it starts feeling pretty kludgy.
Here's a chapter from the GP book on regression.
Here's a chapter on classification, for comparison. Notice that you end up with some complicated approximations or an iterative method.
One nice thing about using GPs for classification, though, is that it gives you a predictive distribution, rather than a simple yes/no classification.
answered Nov 20, 2012 at 3:24
-
2$\begingroup$ +1 GPs are a good alternative to KLR (although KLR often gives better performance because evidence based model selection can go wrong quite easily if there is model mis-specification) and cross-validation is often preferable. $\endgroup$ Commented Nov 20, 2012 at 9:58
$\begingroup$
$\endgroup$
please visit http://www.stanford.edu/~hastie/Papers/svmtalk.pdf
Some conclusions: The classification performance is very similar. Has limiting optimal margin properties. Provides estimates of the class probabilities. Often these are more useful than the classifications. Generalizes naturally to M-class classification through kernel multi-logit regression.
