7
$\begingroup$

I am learning how to use libsvm through sklearn.svm in python. I read here about what happens and why when you change the C value as part of your model. My intuition from what I've learned would be that lower C values would use less support vectors to make a more general classification, while higher C values would use more support vectors to attempt to 'overfit' and account for all outliers.

That is not the case. For an example where I looped through a set of C values like so:

print(c)
model = svm.SVC(kernel='linear', C=c)
model.fit(Xtrain, ytrain)
print("support vectors:", len(model.support_))

I got results:

1.0
support vectors: 1810
10.0
support vectors: 1750
100.0
support vectors: 1626
1000.0
support vectors: 1558

As you can see, as C goes up, the number of support vectors used in the model goes down. Why does this happen?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
8
$\begingroup$

In short, C is the penalty on the slack variables, which measure the degree to which the margin constraints are violated. A training pattern violates the margin constraint if the kernel expansion (i.e. the output of the SVM) has a value between -1 and +1, and all patterns violating this constraint will be support vectors. If you increase C, a greater penalty is put on violation of the constraint, the solution will change to reduce the size of the violations (and hence the number of violations) so the margin is made narrower, and less patterns will fall inside it, so there are fewer support vectors.

The expected error rate depends on both the margin and also on the sum of the slack variables, so in practice generalisation is maximised by a compromise between the two qualities, giving an intermediate value of C.

Improve this answer
$\endgroup$
4
  • 1
    $\begingroup$ Could you clarify what do you mean by "kernel expansion" please @Dikran Marsupial? $\endgroup$
    – Daneel Olivaw
    Commented Mar 28, 2017 at 12:59
  • $\begingroup$ With respect to this answer of yours, you say there that increasing C increases the complexity of the hypothesis class. Can you explain in your answer how the hypothesis class can get more complex if the number of support vectors is reduced? I do not understand that. $\endgroup$
    – kingledion
    Commented Mar 28, 2017 at 13:24
  • 1
    $\begingroup$ @DaneelOlivaw I just mean the output of the support vector machine (before thresholding). $\endgroup$
    – Dikran Marsupial
    Commented Mar 28, 2017 at 13:42
  • 2
    $\begingroup$ @kingledion the number of support vectors is not a particularly good measure of complexity as the magnitude of the weights is also relevant. Vapnik defines the concept of an essential support vector (one that the decision boundary cannot be defined without), but can often find a good approximation with fewer SVs than the algorithm gives. Note also that increasing the complexity of a hypothesis class does not mean increasing the complexity of a particular hypothesis from that class. $\endgroup$
    – Dikran Marsupial
    Commented Mar 28, 2017 at 13:45
0
$\begingroup$

It may also be because you use linear kernel. The Gram matrix X @ X.T may be not full rank, so support vectors cannot overfit.

Try using "poly" kernel with degree 4

Improve this answer
$\endgroup$
1
  • $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Commented Oct 16, 2023 at 6:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.