2
$\begingroup$

I try to train a linear SVM using package e1071, with cost=10 on the following data:

    x <- matrix(c(-0.97, -0.69, -1.14, -0.93, 0.26, 1.14, 0.76, 
            1.32, -0.79, 1.04, -.80, -.97, -1.09, -1.09, 0.63, 
            1.09, 0.92, 1.49, -0.52, 0.34), 10, 2)
    colnames(x) <- c("X1","X2")
    y <- c(-1,-1,-1,-1,1,1,1,1,1,-1)
    svm2 <- svm(x,y,type="C-classification", kernel="linear", 
                cost=10, scale=FALSE)

The data looks like: enter image description here

In theory, point E(0.26,0.63) should also be a support vector. But svm() does not return it as a support vector, but considers point T(1.14,1.09) as SV. This happens also when I change the kernel to "radial".

In theory, E should be a support vector, while point T(1.14, 1.09) not, if I understand correctly the theory.

Might this inconsistency be due to the fact that I have only 10 observations?

$\endgroup$
2
  • $\begingroup$ How this is bad? You have one misclassified case. $\endgroup$
    – Tim
    Aug 29, 2017 at 7:54
  • $\begingroup$ I do not understand why E is not a support vector, given that it lies between the margins. $\endgroup$
    – ralucaGui
    Aug 29, 2017 at 8:11

4 Answers 4

3
$\begingroup$

It sounds like you're trying to apply the intuitions of a hard-margin SVM to the results of a soft-margin SVM and being led astray by the differences.

I can tell you're using a soft-margin SVM because you provided a "cost=10" parameter; the e1071 R library does not appear to support a hard-margin classifier directly, but you can approximate it by using a suitably huge cost parameter, say 1e10.

When the cost is low, a soft-margin SVM will sometimes choose to misclassify one or more points if it means it can get a wider margin. Perhaps surprisingly, they may do this even when the data are truly linear separable and a "perfect" solution could be found. If you increase the cost to 100 or 1,000 it will behave more like the hard-margin version and you will see it correctly classify all training examples.

$\endgroup$
0
$\begingroup$

The main issue is that there exists a linear seperation of both classes without any loss. The correct support vectors should be S, T, A, D with a linear kernel. I do not see any inconsistency with E not beeing a support vector.

The plot illustrates the proper linear seperation of both classes (roughly, not done with svm), support vectors marked in red.

enter image description here

$\endgroup$
2
  • $\begingroup$ Thank you. Can you please let me know what software and which package did you use to get your results? $\endgroup$
    – ralucaGui
    Aug 29, 2017 at 8:09
  • $\begingroup$ I used Python (&matplotlib) to plot your data and I used my basic knowledge of SVM to identify the support vectors. Anyhow I would recommend to use Python and the Scikit-Learn package (well documented with a lot of further information). Although I do not use R, yet I never heard anything bad about it - so it seems to be just as fine. $\endgroup$ Aug 29, 2017 at 8:12
0
$\begingroup$

I think @olooney is correct, but IIRC (it is a long time since I read it) Vapnik's book makes a distinction between support vectors and essential support vectors. Especially for linear classifiers in low dimensions, the optimal decision boundary can be defined using a subset of the support vectors. SVM solvers don't necessarily give you all of the support vectors, just a set of support vectors that is sufficient to define the discriminant. Not sure if that is the case here.

Try retraining the model with point E left out, if the outcome is the same, it is, at most, a non-essential support vector. BTW I don't really understand the use of colours in the diagram.

$\endgroup$
0
$\begingroup$

We can use kernlab in R to get the support vectors with linear kernel:

library(kernlab)
df <- as.data.frame(x)
df$y <- y
m <- ksvm(y~.,data=df,type="C-svc",
     kernel='vanilladot',C=10,prob.model=TRUE)
plot(m,data=df, xlim=c(-1.2,1.2), ylim=c(-1.2,1.2))

Note that the separating hyperplane found is different from the one drawn and the highlighted black points represent 4 support vectors. As expected, points near the decision surface are chosen to be the support vectors.

enter image description here

With RBF kernel (rbfdot) and keeping other hyper-parameters fixed, the decision surface becomes non-linear and SVM outputs more (7) support vectors, as shown below:

enter image description here

Changing (increasing) the regularization hyper-parameter $C$ (keeping other hyper-parameters fixed and using RBF kernel) will result in a different decision boundary and change (decrease) the number of support vectors (back to 4), as shown below.

enter image description here

The following animation shows how the decision surface (the orientation of the decision hyperplance along with the margins) and the support vectors change with different value of the hyper-parameter C and kernel. As expected, with soft-margin SVM (for low values of $C$), we need more support vectors, to allow more noisy data, when compared to hard-margin SVM, that requires less number of support vectors (with narrower margins).

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.