Is support vector machine with linear kernel the same as a soft margin classifier?
3 Answers
$\begingroup$
$\endgroup$
5
The idea of soft margin classifier means that one allows a number of misclassifications on the wrong part of the margin. This has nothing to do with the used kernel. Using a linear kernel or other type of kernel only affects in which kind of space the separating margin is searched. All SVMs uses soft margin, but one can push the soft margin behavior towards a hard margin behavior but imposing a very large penalty on errors.
[Later edit:]
Linear SVMs either for regression or for classification have a special characteristic. The numerical computations are simpler because one can compress the original result of the algorithm into a much simpler formula (due to it's linearity). This is why often implementations for SVMs and SVCs have sometimes separate implementations for the linear kernel.
-
$\begingroup$ Thank you very much for your answer. So I have the task to train an Support Vector Classifier on my training data , an svm with RBF and svm with polynomial kernel. If I use the 'svmLinear' method in caret for the SVC am I correct or should I look for another procedure? Cause from the question I understand that although SVMs are based on soft margin, I have to follow a different procedure for the SVC... $\endgroup$– KkkApr 5, 2021 at 18:01
-
1$\begingroup$ SVM is a generic name for the algorithm. When it does classification sometimes it is named SVC and when it does regression sometimes it is named SRV. You can't build a SVC/SRV with RBF or poly kernels with the svmLinear. That implementation is specific for linear kernel. $\endgroup$– rapaioApr 5, 2021 at 18:28
-
-
$\begingroup$ Just found this, the question mentions support vector classifier and then solves it as svm with linear kernel and compares it with the svm radial and polynomial $\endgroup$– KkkApr 5, 2021 at 18:30
-
1$\begingroup$ As I said, svm is sometimes named in implementations as svc (in sklearn for example) when it is used for classiffication and sometimes not. The same for regression. $\endgroup$– rapaioApr 5, 2021 at 18:32
$\begingroup$
$\endgroup$
2
No. The two are independent of each other.
The kernel $K(x, z)$ is the generalisation of the scalar (dot) product, $x \cdot z$. We often choose $K(x, z)$ to be a non-linear function (polynomial, RBF, etc.). If not, i.e. if $K(x, z) = x \cdot z$, we say that the kernel is linear.
Independently of the kernel, a classifier can use a hard or a soft margin. A hard-margin classifier requires the classes to be linearly separable in the kernel-induced feature space. For the linear kernel this is the same as simply saying that the classes are linearly separable, but a non-linear kernel can also transform non-separable data into separable ones.
A soft-margin classifier can tolerate misclassification of some observations, making it suitable for training even if the data are not separable.
So, the choice of the kernel and the choice between the hard or soft margin are two distinct choices you need to make. In practice we almost always use soft margin, but control for its "softness" using the parameter $C$. As $C \rightarrow \infty$, the margin approaches the hard margin.
-
-
$\begingroup$ You can express your thanks by upvoting and/or accepting the answer. $\endgroup$– Igor F.Apr 6, 2021 at 4:55
$\begingroup$
$\endgroup$
4
From what I have understood, and also from what the author of this book is telling here, a linear kernel is another representation of the support vector classifier. As he adds in the end, when the support vector classifier is combined with a non-linear kernel (polynomial/RBF) we have a support vector machine. SVCs and SVMs use a soft margin whereas only a Maximal Margin Classifier uses a hard margin, which is different from a support vector classifier.
-
$\begingroup$ Just bear in mind that the authors you quote (James, Witten, Hastie, Tibshirani) are all statisticians. If you want to be precise about the terminology, you should consult the original papers by Vapnik et al., or books written by (co-)developers of the SVMs (incl. Vapnik). $\endgroup$– Igor F.Apr 6, 2021 at 6:29
-
$\begingroup$ So because they are statisticians their explanation is imprecise? rstudio-pubs-static.s3.amazonaws.com/… Also from what I mentioned before, in the questions for the AWS certification, they refer to SVC as linear kernel as well. If you have a paper of the original author that contradicts these, please edit your answer to quote their sayings. $\endgroup$– KkkApr 6, 2021 at 12:09
-
$\begingroup$ I doubt statisticians, as (reluctant) users of SVMs, are a higher authority (or even claim to be an authority) regarding the terminology than the people who invented them. You can take a look at Burges’ SVM Tutorial, sections 3 and 4: kernel-machines.org/publications/Burges98. $\endgroup$– Igor F.Apr 7, 2021 at 11:30
-
$\begingroup$ @IgorF. Hastie and Tibshirani are one of the most influential people in machine learning. They are inventors of lasso, elastic net, gradient boosting, generalized additive models and other methods, Tibshirani did his PhD with Efron, the inventor of bootstrap and Hastie with Breiman, the inventor of random forrest. I am sure they are qualified enough to talk about SVM. Vapnik is also a statistician, not that it would matter regarding his authority $\endgroup$– rep_hoApr 7 at 9:25