Does SVM perform poorly with too few observations

Question

I try to train a linear SVM using package e1071, with cost=10 on the following data:

    x <- matrix(c(-0.97, -0.69, -1.14, -0.93, 0.26, 1.14, 0.76, 
            1.32, -0.79, 1.04, -.80, -.97, -1.09, -1.09, 0.63, 
            1.09, 0.92, 1.49, -0.52, 0.34), 10, 2)
    colnames(x) <- c("X1","X2")
    y <- c(-1,-1,-1,-1,1,1,1,1,1,-1)
    svm2 <- svm(x,y,type="C-classification", kernel="linear", 
                cost=10, scale=FALSE)

The data looks like:

In theory, point E(0.26,0.63) should also be a support vector. But svm() does not return it as a support vector, but considers point T(1.14,1.09) as SV. This happens also when I change the kernel to "radial".

In theory, E should be a support vector, while point T(1.14, 1.09) not, if I understand correctly the theory.

Might this inconsistency be due to the fact that I have only 10 observations?

Answer 1

It sounds like you're trying to apply the intuitions of a hard-margin SVM to the results of a soft-margin SVM and being led astray by the differences.

I can tell you're using a soft-margin SVM because you provided a "cost=10" parameter; the e1071 R library does not appear to support a hard-margin classifier directly, but you can approximate it by using a suitably huge cost parameter, say 1e10.

When the cost is low, a soft-margin SVM will sometimes choose to misclassify one or more points if it means it can get a wider margin. Perhaps surprisingly, they may do this even when the data are truly linear separable and a "perfect" solution could be found. If you increase the cost to 100 or 1,000 it will behave more like the hard-margin version and you will see it correctly classify all training examples.

Answer 2

The main issue is that there exists a linear seperation of both classes without any loss. The correct support vectors should be S, T, A, D with a linear kernel. I do not see any inconsistency with E not beeing a support vector.

The plot illustrates the proper linear seperation of both classes (roughly, not done with svm), support vectors marked in red.

Answer 3

I think @olooney is correct, but IIRC (it is a long time since I read it) Vapnik's book makes a distinction between support vectors and essential support vectors. Especially for linear classifiers in low dimensions, the optimal decision boundary can be defined using a subset of the support vectors. SVM solvers don't necessarily give you all of the support vectors, just a set of support vectors that is sufficient to define the discriminant. Not sure if that is the case here.

Try retraining the model with point E left out, if the outcome is the same, it is, at most, a non-essential support vector. BTW I don't really understand the use of colours in the diagram.

Answer 4

We can use kernlab in R to get the support vectors with linear kernel:

library(kernlab)
df <- as.data.frame(x)
df$y <- y
m <- ksvm(y~.,data=df,type="C-svc",
     kernel='vanilladot',C=10,prob.model=TRUE)
plot(m,data=df, xlim=c(-1.2,1.2), ylim=c(-1.2,1.2))

Note that the separating hyperplane found is different from the one drawn and the highlighted black points represent 4 support vectors. As expected, points near the decision surface are chosen to be the support vectors.

With RBF kernel (rbfdot) and keeping other hyper-parameters fixed, the decision surface becomes non-linear and SVM outputs more (7) support vectors, as shown below:

Changing (increasing) the regularization hyper-parameter $C$ (keeping other hyper-parameters fixed and using RBF kernel) will result in a different decision boundary and change (decrease) the number of support vectors (back to 4), as shown below.

The following animation shows how the decision surface (the orientation of the decision hyperplance along with the margins) and the support vectors change with different value of the hyper-parameter C and kernel. As expected, with soft-margin SVM (for low values of $C$), we need more support vectors, to allow more noisy data, when compared to hard-margin SVM, that requires less number of support vectors (with narrower margins).