How to construct the feature weight vector (or decision boundary) from a linear SVM classifier with scikit?

Question

I use the following code to train an SVM classifier:

clf = svm.SVC(kernel='linear')
clf.fit(train_mat, train_labels)

It fits the data and saves the info in the clf object. Now I know how theoretically the w vector is constructed in the formula. It is a sum over all support vectors multiplied by their labels and the corresponding alpha values. Problem is, I can't seem to find this info in clf. I have a _coef attribute but I am not sure if it is what I am looking for.

Documentation in scikit is simply lacking. I couldn't find the info there. Does anyone know how to do that? I need the w vector to check the weights given to each of the features and see which ones are the most valuable.

Answer 1

In your clf,

coef_ are the weights assigned to the features; (Note it only works for linear SVM)

support_vectors_ and support_ are the support vectors and the corresponding index;

dual_coef_ is the coefficients of the support vector in the decision function; and

intercept_ is the bias in decision function.

In linear SVM, $w^Tx+b=0$ is the decision boundary, and $w$ is the coefficients of the support vectors, $b$ is the bias, all defined above.

The document reference: http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html

---

For an RBF SVM case, original data space is transformed into another high-dimensional space. So the weights coefficients are not directly related to the input space. I think that's why coef_ cannot be viewed as the weights in your original input space. By the way, I'm not sure whether coef_ is the weight in the transformed space feature, I guess it's not, as the RBF space is actually infinite dimensional.

I suggest you preprocess the data before SVM implementation since SVM is heavily influenced by feature scale variances.

1. Standard normalization of the data;

2. Decorrelation sigma^(-1/2)*X where sigma = cov(X).

---

Yet you may need to do some calculation by yourself to get the decision function.

1. Compute the feature vector v from your data point under test. The length of v is supposed to be the same as the rows of support_vectors_.

2. For each row i in support_vectors_, compute the Euclidean distance d[i] = numpy.linalg.norm(support_vectors_[i,] - v) .

3. t[i] = exp{-gamma *d[i].^2} where gamma is the RBF parameter.

4. Sum up dual_coef_[i] * t[i] over all i, then plus intercept_. This will be the decision function.

Answer 2

from sklearn import datasets
from sklearn import svm
import numpy as np

data = datasets.load_digits(n_class=2)
clf = svm.SVC(kernel="linear")
clf = clf.fit(data.data,data.target)
print(np.matmul(clf.dual_coef_,clf.support_vectors_)) 
print(clf.intercept_)

This should print a numpy array of your weights and b.

You can double check this through the coef_ variable that is only available in linear kernel:

print(clf.coef_)

This should print the weights identically twice.