# How does one interpret SVM feature weights?

I am trying to interpret the variable weights given by fitting a linear SVM.

(I'm using scikit-learn):

from sklearn import svm

svm = svm.SVC(kernel='linear')

svm.fit(features, labels)
svm.coef_


I cannot find anything in the documentation that specifically states how these weights are calculated or interpreted.

Does the sign of the weight have anything to do with class?

-
Are you using sklearn.svm.LinearSVC() ? –  Bitwise Oct 12 '12 at 1:36
no, sorry. svm = sklearn.svm.SVC(kernel='linear') –  Austin Oct 12 '12 at 16:22
I don't see documentation for svm.weights, do you mean svm.coef_ ? –  Bitwise Oct 12 '12 at 17:01
Yes that too :) –  Austin Oct 12 '12 at 20:10

For a general kernel it is difficult to interpret the SVM weights, however for the linear SVM there actually is a useful interpretation:

1) Recall that in linear SVM, the result is a hyperplane that separates the classes as best as possible. The weights represent this hyperplane, by giving you the coordinates of a vector which is orthogonal to the hyperplane - these are the coefficients given by svm.coef_. Let's call this vector w.

2) What can we do with this vector? It's direction gives us the predicted class, so if you take the dot product of any point with the vector, you can tell on which side it is: if the dot product is positive, it belongs to the positive class, if it is negative it belongs to the negative class.

3) Finally, you can even learn something about the importance of each feature. This is my own interpretation so convince yourself first. Let's say the svm would find only one feature useful for separating the data, then the hyperplane would be orthogonal to that axis. So, you could say that the absolute size of the coefficient relative to the other ones gives an indication of how important the feature was for the separation. For example if only the first coordinate is used for separation, w will be of the form (x,0) where x is some non zero number and then |x|>0.

-
Point 3 is the basis for the RFE algorithm using the weight vector of a linear SVM for feature (gene) selection: See Guyon axon.cs.byu.edu/Dan/778/papers/Feature%20Selection/guyon2.pdf –  B_Miner Oct 13 '12 at 0:44
@B_Miner thanks! I was worried that since I thought of this on my own it might be wrong (I am not from "pure" CS) - but I guess it is correct. –  Bitwise Oct 13 '12 at 1:34
What is the meaning of the direction of the orthogonal vector if it is separating both classes? Does it have something to do with the contribution of the separating hyperplane to the overall probability of class prediction? –  Austin Oct 13 '12 at 3:48
The documentation is pretty complete: for the multiclass case, SVC which is based on the libsvm library uses the one-vs-one setting. In the case of a linear kernel, n_classes * (n_classes - 1) / 2 individual linear binary models are fitted for each possible class pair. Hence the aggregate shape of all the primal parameters concatenated together is [n_classes * (n_classes - 1) / 2, n_features] (+ [n_classes * (n_classes - 1) / 2 intercepts in the intercept_ attribute).
For the binary linear problem, plotting the separating hyperplane from the coef_ attribute is done in this example.