1
$\begingroup$

I'm currently playing around with support vector machines in Scikit Learn and I've come across some unusual behaviour. For a basic simulated dataset, I've trained an SVC estimator (with linear kernel), found the support vectors and computed the decision function scores.

It's those scores that are confusing me; according to Scikit, some of the scores are greater than 1 or less than -1; isn't this odd? The margins in a linear support vector classifier should be $x^{T}w+b = \pm 1$, so any examples with score outside $[-1,1]$ shouldn't be a support vector. I've attached my code below - what's going wrong?

from sklearn.svm import SVC

clf = SVC(C = 1.0, kernel = 'linear', class_weight = 'balanced',tol=1e-10)

clf.fit(X,Y)

w = clf.coef_
b = clf.intercept_

sv = clf.support_vectors_
print(w,b)
print('Number of support vectors:', sv.shape[0])
print(clf.decision_function(sv))

Here is the output:

[[1.28546328 1.08848812]] [-0.35044999]
Number of support vectors: 44
array([-0.57011282, -0.59506839, -0.53074738, -0.36055916, -0.69388818,
         0.28055111,  1.39559261,  0.26911971, -0.90759079, -0.93444484,
         2.37910343,  0.13621966, -0.86860512, -0.93672977,  0.85133397,
        -0.85497661, -0.78276458,  0.3908212 , -0.35721162, -1.00000001,
         2.07987489, -0.89293624,  0.37559497,  0.3603185 ,  0.35699434,
         0.23356079,  0.4274206 ,  0.77953069,  0.50193484,  0.99999995,
         0.7034846 ,  0.68110044, -2.00881614,  0.17607163,  0.99999989,
         0.40207295,  0.81320519, -1.31492698,  0.84101353,  0.3534314 ,
         0.39314117, -1.62172733, -2.41029845, -0.56251937])
$\endgroup$

2 Answers 2

1
$\begingroup$

Misclassified points will always be support vectors, even when they are "badly misclassified" and lie beyond the margin, with decision function scores outside of $[-1, 1]$.

$\endgroup$
0
$\begingroup$

SVC.decision_function returns values in $\mathbb{R}$ and is defined as: $$ f(x) = \sum_{i \in SV} y_i \alpha_i \langle x, x_i \rangle + b $$ You expect to get $1$ or $-1$ when predicting classes, not scores, with SVC.predict which is $\text{sign}(f(x))$. So, I believe everything is OK.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.