# How to construct the feature weight vector (or decision boundary) from a linear SVM classifier with scikit?

I use the following code to train an SVM classifier:

clf = svm.SVC(kernel='linear')
clf.fit(train_mat, train_labels)


It fits the data and saves the info in the clf object. Now I know how theoretically the w vector is constructed in the formula. It is a sum over all support vectors multiplied by their labels and the corresponding alpha values. Problem is, I can't seem to find this info in clf. I have a _coef attribute but I am not sure if it is what I am looking for.

Documentation in scikit is simply lacking. I couldn't find the info there. Does anyone know how to do that? I need the w vector to check the weights given to each of the features and see which ones are the most valuable.

## 2 Answers

In your clf,

coef_ are the weights assigned to the features; (Note it only works for linear SVM)

support_vectors_ and support_ are the support vectors and the corresponding index;

dual_coef_ is the coefficients of the support vector in the decision function; and

intercept_ is the bias in decision function.

In linear SVM, $w^Tx+b=0$ is the decision boundary, and $w$ is the coefficients of the support vectors, $b$ is the bias, all defined above.

The document reference: http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html

For an RBF SVM case, original data space is transformed into another high-dimensional space. So the weights coefficients are not directly related to the input space. I think that's why coef_ cannot be viewed as the weights in your original input space. By the way, I'm not sure whether coef_ is the weight in the transformed space feature, I guess it's not, as the RBF space is actually infinite dimensional.

I suggest you preprocess the data before SVM implementation since SVM is heavily influenced by feature scale variances.

1. Standard normalization of the data;

2. Decorrelation sigma^(-1/2)*X where sigma = cov(X).

Yet you may need to do some calculation by yourself to get the decision function.

1. Compute the feature vector v from your data point under test. The length of v is supposed to be the same as the rows of support_vectors_.

2. For each row i in support_vectors_, compute the Euclidean distance d[i] = numpy.linalg.norm(support_vectors_[i,] - v) .

3. t[i] = exp{-gamma *d[i].^2} where gamma is the RBF parameter.

4. Sum up dual_coef_[i] * t[i] over all i, then plus intercept_. This will be the decision function.

• I thought so, but couldnt understand it from the docs. The reason I was not sure about it is because I see a strange phenomenon in my coef_ vector. I have only 10 featurs, with some pairs highly positively correlated. In some pairs one feature gets a positive non-negligible weight, and the other gets a non-negligible negative weight. because that seemed very unreasonable to me, I thought that the coef_ has to be involved with other clf.att to construct the weight vector. Can you try to explain this? Feb 14, 2014 at 8:29
• @idoda, coef_ only works for linear kernel, but not rbf. Please see my update in the answers. Hope it helps, thanks Feb 14, 2014 at 17:19
• I didnt understand your comment (why would you talk about rbf when im asking about linear kernel) until I took a second look at the code I published. fix it - I was referring to a linear kernel just as I wrote in the headline. But thanks for the clarification for the rbf case anyway ! Feb 15, 2014 at 9:45
• when changing it to linear, are you still facing the weight sign problem? Yet I would suggest you decorrelate your data before svm. Thanks Feb 15, 2014 at 17:00
from sklearn import datasets
from sklearn import svm
import numpy as np

data = datasets.load_digits(n_class=2)
clf = svm.SVC(kernel="linear")
clf = clf.fit(data.data,data.target)
print(np.matmul(clf.dual_coef_,clf.support_vectors_))
print(clf.intercept_)


This should print a numpy array of your weights and b.

You can double check this through the coef_ variable that is only available in linear kernel:

print(clf.coef_)


This should print the weights identically twice.