2
$\begingroup$

I use the following code to train an SVM classifier:

clf = svm.SVC(kernel='linear')
clf.fit(train_mat, train_labels)

It fits the data and saves the info in the clf object. Now I know how theoretically the w vector is constructed in the formula. It is a sum over all support vectors multiplied by their labels and the corresponding alpha values. Problem is, I can't seem to find this info in clf. I have a _coef attribute but I am not sure if it is what I am looking for.

Documentation in scikit is simply lacking. I couldn't find the info there. Does anyone know how to do that? I need the w vector to check the weights given to each of the features and see which ones are the most valuable.

$\endgroup$
5
$\begingroup$

In your clf,

coef_ are the weights assigned to the features; (Note it only works for linear SVM)

support_vectors_ and support_ are the support vectors and the corresponding index;

dual_coef_ is the coefficients of the support vector in the decision function; and

intercept_ is the bias in decision function.

In linear SVM, $w^Tx+b=0$ is the decision boundary, and $w$ is the coefficients of the support vectors, $b$ is the bias, all defined above.

The document reference: http://scikit-learn.org/stable/modules/generated/sklearn.svm.SVC.html


For an RBF SVM case, original data space is transformed into another high-dimensional space. So the weights coefficients are not directly related to the input space. I think that's why coef_ cannot be viewed as the weights in your original input space. By the way, I'm not sure whether coef_ is the weight in the transformed space feature, I guess it's not, as the RBF space is actually infinite dimensional.

I suggest you preprocess the data before SVM implementation since SVM is heavily influenced by feature scale variances.

  1. Standard normalization of the data;

  2. Decorrelation sigma^(-1/2)*X where sigma = cov(X).


Yet you may need to do some calculation by yourself to get the decision function.

  1. Compute the feature vector v from your data point under test. The length of v is supposed to be the same as the rows of support_vectors_.

  2. For each row i in support_vectors_, compute the Euclidean distance d[i] = numpy.linalg.norm(support_vectors_[i,] - v) .

  3. t[i] = exp{-gamma *d[i].^2} where gamma is the RBF parameter.

  4. Sum up dual_coef_[i] * t[i] over all i, then plus intercept_. This will be the decision function.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ I thought so, but couldnt understand it from the docs. The reason I was not sure about it is because I see a strange phenomenon in my coef_ vector. I have only 10 featurs, with some pairs highly positively correlated. In some pairs one feature gets a positive non-negligible weight, and the other gets a non-negligible negative weight. because that seemed very unreasonable to me, I thought that the coef_ has to be involved with other clf.att to construct the weight vector. Can you try to explain this? $\endgroup$ – idoda Feb 14 '14 at 8:29
  • $\begingroup$ @idoda, coef_ only works for linear kernel, but not rbf. Please see my update in the answers. Hope it helps, thanks $\endgroup$ – lennon310 Feb 14 '14 at 17:19
  • $\begingroup$ I didnt understand your comment (why would you talk about rbf when im asking about linear kernel) until I took a second look at the code I published. fix it - I was referring to a linear kernel just as I wrote in the headline. But thanks for the clarification for the rbf case anyway ! $\endgroup$ – idoda Feb 15 '14 at 9:45
  • $\begingroup$ when changing it to linear, are you still facing the weight sign problem? Yet I would suggest you decorrelate your data before svm. Thanks $\endgroup$ – lennon310 Feb 15 '14 at 17:00
0
$\begingroup$
from sklearn import datasets
from sklearn import svm
import numpy as np

data = datasets.load_digits(n_class=2)
clf = svm.SVC(kernel="linear")
clf = clf.fit(data.data,data.target)
print(np.matmul(clf.dual_coef_,clf.support_vectors_)) 
print(clf.intercept_)

This should print a numpy array of your weights and b.

You can double check this through the coef_ variable that is only available in linear kernel:

print(clf.coef_)

This should print the weights identically twice.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.