Why does the linear SVM give a lot of support vectors?

I simulate a simple linear setup:

n = 1000
X = runif(n)
Y = runif(n)

ind = X + 2*Y < 1
ind[ind == TRUE] = runif(sum(ind)) < 1
plot(X,Y,col = ind + 1)


Which gives

The svm() functcion from e1071 performs very well but it gives me a lot of vectors.

Call:
svm(formula = ind ~ X + Y, type = "C-classification", kernel = "linear")

Parameters:
SVM-Type:  C-classification
SVM-Kernel:  linear
cost:  1

Number of Support Vectors:  99


Can you please tell me what and how should I tune to get one vector (or just a few)?

• Increase the value of the cost parameter to have less support vectors. If the kernel you use is linear, gamma is not used. Nov 24, 2014 at 12:49
• Thanks a lot, that works! gamma was there from my another tests, I will update the code. Nov 24, 2014 at 12:53
• It would be fun to use cross validation to see what is the optimal value for this cost parameter ! Nov 24, 2014 at 12:57
• For sure. In fact in the e1071 package there is a tune.svm procedure, which takes a given set of parameters and does CV automatically, but this is a bit different story. Thanks for helping with this one :) Nov 24, 2014 at 13:03
• In my experience, tuning the regularisation parameter using CV generally results in models with lots of support vectors. I would view the sparsity of SVM as a convenient by-product, but nothing more. If you really want a sparse model, try something like L1 regularised logistic regression. Nov 24, 2014 at 14:52

Any linear SVM can be summarized as a single vector in input space:

$$f(\mathbf{z}) = \sum_{i \in SV} y_i \alpha_i \mathbf{x}_i^T\mathbf{z} +b,$$

can be rewritten as:

$$f(\mathbf{z}) = \mathbf{w}^T\mathbf{z} + b,$$

with

$$w[k] = \sum_{i\in SV} y_i \alpha_i x_i[k], \quad k=1..d$$

The amount of support vectors that actually form the model is not that relevant for a linear SVM, except for prediction speed (the above comment applies).

The problem here is that e1071 apparently uses LIBSVM instead of LIBLINEAR for linear SVM's. LIBSVM doesn't turn a linear model into a single vector in input space.