2
$\begingroup$

I simulate a simple linear setup:

n = 1000
X = runif(n)
Y = runif(n)

ind = X + 2*Y < 1
ind[ind == TRUE] = runif(sum(ind)) < 1
plot(X,Y,col = ind + 1)

Which gives

linear split

The svm() functcion from e1071 performs very well but it gives me a lot of vectors.

Call:
svm(formula = ind ~ X + Y, type = "C-classification", kernel = "linear")

Parameters:
   SVM-Type:  C-classification 
 SVM-Kernel:  linear 
       cost:  1 

Number of Support Vectors:  99

Can you please tell me what and how should I tune to get one vector (or just a few)?

$\endgroup$
5
  • 2
    $\begingroup$ Increase the value of the cost parameter to have less support vectors. If the kernel you use is linear, gamma is not used. $\endgroup$ Nov 24, 2014 at 12:49
  • $\begingroup$ Thanks a lot, that works! gamma was there from my another tests, I will update the code. $\endgroup$ Nov 24, 2014 at 12:53
  • $\begingroup$ It would be fun to use cross validation to see what is the optimal value for this cost parameter ! $\endgroup$ Nov 24, 2014 at 12:57
  • $\begingroup$ For sure. In fact in the e1071 package there is a tune.svm procedure, which takes a given set of parameters and does CV automatically, but this is a bit different story. Thanks for helping with this one :) $\endgroup$ Nov 24, 2014 at 13:03
  • 1
    $\begingroup$ In my experience, tuning the regularisation parameter using CV generally results in models with lots of support vectors. I would view the sparsity of SVM as a convenient by-product, but nothing more. If you really want a sparse model, try something like L1 regularised logistic regression. $\endgroup$ Nov 24, 2014 at 14:52

1 Answer 1

3
$\begingroup$

Any linear SVM can be summarized as a single vector in input space:

$$f(\mathbf{z}) = \sum_{i \in SV} y_i \alpha_i \mathbf{x}_i^T\mathbf{z} +b,$$

can be rewritten as:

$$f(\mathbf{z}) = \mathbf{w}^T\mathbf{z} + b,$$

with

$$w[k] = \sum_{i\in SV} y_i \alpha_i x_i[k], \quad k=1..d$$

The amount of support vectors that actually form the model is not that relevant for a linear SVM, except for prediction speed (the above comment applies).

The problem here is that e1071 apparently uses LIBSVM instead of LIBLINEAR for linear SVM's. LIBSVM doesn't turn a linear model into a single vector in input space.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.