1
$\begingroup$

My question is about the output in sklearn.svm.SVC function in Python. Apologies for a software context question but I believe a good number of those who have learnt SVM will have come across this function.

In the sklearn documentation section 1.4.7.1, it is stated that 'These parameters can be accessed through the attributes dual_coef_ which holds the product $y_i \alpha_i$. The array returned by dual_coef_ is of shape (n_classes-1, n_sv).

As I understand, the support vectors are those data points whose $\alpha_i > 0$, thus they will make an impact on the decision boundary which has the form \begin{align*} f(x) = \sum_i \alpha_i K(x_i, x) \end{align*} Those non-support vectors have their $\alpha_i = 0$.

When I fit the following model however the output array in dual_coef_ does contain zeros. What is the explanation for that? Thanks in advance.

svm= SVC(C= 10, kernel='rbf', gamma= 'scale', decision_function_shape= 'ovo', max_iter=-1, probability=True)
svm_mod= svm.fit(xtr, ytr)
print('yi * alpha_i: \n', svm_mod.dual_coef_)
print('Number of support vectors of each class:', svm_mod.n_support_)

# Output
yi * alpha_i: 
 [[  4.58635603  -0.          -0.          -0.          -0.
   -4.58635603  -0.          -0.          -0.          -0.
   -0.          -0.          -0.39551576  -0.          -0.
   -1.10882587  -0.          -0.          -0.        ]
 [  1.50434163  10.          10.          10.           3.00420504
    0.          10.          10.          10.          10.
  -10.          -3.00420504  -0.         -10.         -10.
  -10.         -10.         -10.         -10.        ]]

Number of support vectors of each class: [1 9 9]
$\endgroup$
1
$\begingroup$

The issue is that you have a multiclass model. Each column represents a support vector for at least one of the binary sub-classifiers. So the non-zero property that needs to hold is that each column contains at least one nonzero entry. From (near the end of) section 1.4.1.1:

The shape of dual_coef_ is (n_classes-1, n_SV) with a somewhat hard to grasp layout. The columns correspond to the support vectors involved in any of the n_classes * (n_classes - 1) / 2 “one-vs-one” classifiers. Each of the support vectors is used in n_classes - 1 classifiers. The n_classes - 1 entries in each row [sic? column?] correspond to the dual coefficients for these classifiers.

There's an example after that; I can't say I found it particularly illuminating (and I think it's suffering the same transposition I noted in the quote), but perhaps you will.


I've had a chance to produce an example notebook. It uses the iris dataset, with just two features for visualization; in the plots below, class 0 is purple, class 1 is green, and class 2 is yellow. The support vectors are circled; there are in particular 3 support vectors from class 0, and the first three columns of dual_coef_ are:

Class 0
support vectors    dual coefs
[5.0, 3.0]          2.3,  0.0
[4.5, 2.3]         10.0,  8.0
[5.4, 3.4]         10.0,  5.6

The first dual coefficient for each support vector corresponds to the one-vs-one classifier of class 0 vs class 1, while the second one corresponds to the class 0 vs class 2. We get evidence for that here: the zero coefficient for the first vector indicates it is outside the margin of the class 0 vs class 2 classifier.

class 0 vs class 1 SVM plot

class 0 vs class 2 SVM plot

The support vectors for classes 1 and 2 primarily have dual coefficients listed as zero first then something nonzero. Here are the only exceptions to that:

Class 1:
[5.4, 3.0]        -10.0,   0.0
[5.2, 2.7]         -2.3,   0.0
[4.9, 2.4]        -10.0,   0.0

Class 2:
[5.6, 2.8]         -3.6, -10.0
[4.9, 2.5]        -10.0, -10.0

In the top plots, note the green points within the margin are these three, and in the second plot, the yellow points within the margin are these two.

Finally, below is the last ovo model. There are obviously many more support vectors, which I won't list. But note that the class 1 support vectors listed above are all outside this margin, and have zeros as the second dual coefficients; however, the class 2 support vectors above are both on the wrong side of the separating plane (not in the margin, but still misclassified!) and have nonzero dual coefficients in the second slot.

class 1 vs class 2 SVM plot

$\endgroup$
3
  • $\begingroup$ Hi thanks for your answer. Just want to confirm my understanding: the rows represent K-1 of the K classes, so if one particular column is completely zeros it means the corresponding support vector is used in the class that has been left out? $\endgroup$
    – siegfried
    May 13 at 0:58
  • $\begingroup$ I am also not very sure about how we fit KC2 classifiers and then each support vector is used in K-1 classifiers. Could you clarify what is the logic here? $\endgroup$
    – siegfried
    May 13 at 0:59
  • $\begingroup$ I think (and I'll try to run a test on the iris dataset) each support vector is associated with its own class, and so appears in the K-1 classifiers of its class vs another; e.g., a support vector of class 1 appears in the 0v1, 1v2, 1v3, ..., 1v(K-1) classifiers. This would mean that the rows of dual_coef_ don't really represent anything consistent. $\endgroup$ May 13 at 14:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.