36
$\begingroup$

I am trying to interpret the variable weights given by fitting a linear SVM.

(I'm using scikit-learn):

from sklearn import svm

svm = svm.SVC(kernel='linear')

svm.fit(features, labels)
svm.coef_

I cannot find anything in the documentation that specifically states how these weights are calculated or interpreted.

Does the sign of the weight have anything to do with class?

$\endgroup$
49
$\begingroup$

For a general kernel it is difficult to interpret the SVM weights, however for the linear SVM there actually is a useful interpretation:

1) Recall that in linear SVM, the result is a hyperplane that separates the classes as best as possible. The weights represent this hyperplane, by giving you the coordinates of a vector which is orthogonal to the hyperplane - these are the coefficients given by svm.coef_. Let's call this vector w.

2) What can we do with this vector? It's direction gives us the predicted class, so if you take the dot product of any point with the vector, you can tell on which side it is: if the dot product is positive, it belongs to the positive class, if it is negative it belongs to the negative class.

3) Finally, you can even learn something about the importance of each feature. This is my own interpretation so convince yourself first. Let's say the svm would find only one feature useful for separating the data, then the hyperplane would be orthogonal to that axis. So, you could say that the absolute size of the coefficient relative to the other ones gives an indication of how important the feature was for the separation. For example if only the first coordinate is used for separation, w will be of the form (x,0) where x is some non zero number and then |x|>0.

$\endgroup$
  • 2
    $\begingroup$ Point 3 is the basis for the RFE algorithm using the weight vector of a linear SVM for feature (gene) selection: See Guyon axon.cs.byu.edu/Dan/778/papers/Feature%20Selection/guyon2.pdf $\endgroup$ – B_Miner Oct 13 '12 at 0:44
  • $\begingroup$ @B_Miner thanks! I was worried that since I thought of this on my own it might be wrong (I am not from "pure" CS) - but I guess it is correct. $\endgroup$ – Bitwise Oct 13 '12 at 1:34
  • $\begingroup$ What is the meaning of the direction of the orthogonal vector if it is separating both classes? Does it have something to do with the contribution of the separating hyperplane to the overall probability of class prediction? $\endgroup$ – Austin Richardson Oct 13 '12 at 3:48
  • $\begingroup$ To elaborate on whether the sign of the weight relates to class (in the linear case) - it depends on the features. For example, if the predictive features take only nonnegative ($\geq 0$) values, then negative weights contribute to a negative classification of data points. $\endgroup$ – Kdawg Jan 18 '16 at 9:44
  • $\begingroup$ @B_Miner , I think you meant to link to this paper rather than the other one by Guyon. $\endgroup$ – ijoseph Mar 14 '18 at 0:00
11
$\begingroup$

The documentation is pretty complete: for the multiclass case, SVC which is based on the libsvm library uses the one-vs-one setting. In the case of a linear kernel, n_classes * (n_classes - 1) / 2 individual linear binary models are fitted for each possible class pair. Hence the aggregate shape of all the primal parameters concatenated together is [n_classes * (n_classes - 1) / 2, n_features] (+ [n_classes * (n_classes - 1) / 2 intercepts in the intercept_ attribute).

For the binary linear problem, plotting the separating hyperplane from the coef_ attribute is done in this example.

If you want the details on the meaning of the fitted parameters, especially for the non linear kernel case have a look at the mathematical formulation and the references mentioned in the documentation.

$\endgroup$
  • 1
    $\begingroup$ In the documentation of Sklearn, the coef_ attribute is of shape = [n_class-1, n_features]. I believe it's a mistake. $\endgroup$ – Naomi Mar 19 '18 at 13:34
6
$\begingroup$

I am trying to interpret the variable weights given by fitting a linear SVM.

A good way to understand how the weights are calculated and how to interpret them in the case of linear SVM is to perform the calculations by hand on a very simple example.

Example

Consider the following dataset which is linearly separable

import numpy as np
X = np.array([[3,4],[1,4],[2,3],[6,-1],[7,-1],[5,-3]] )
y = np.array([-1,-1, -1, 1, 1 , 1 ])

SVM simple

Solving the SVM problem by inspection

By inspection we can see that the boundary line that separates the points with the largest "margin" is the line $x_2 = x_1 - 3$. Since the weights of the SVM are proportional to the equation of this decision line (hyperplane in higher dimensions) using $w^T x + b = 0$ a first guess of the parameters would be

$$ w = [1,-1] \ \ b = -3$$

SVM theory tells us that the "width" of the margin is given by $ \frac{2}{||w||}$. Using the above guess we would obtain a width of $\frac{2}{\sqrt{2}} = \sqrt{2}$. which, by inspection is incorrect. The width is $4 \sqrt{2}$

Recall that scaling the boundary by a factor of $c$ does not change the boundary line, hence we can generalize the equation as

$$ cx_1 - xc_2 - 3c = 0$$ $$ w = [c,-c] \ \ b = -3c$$

Plugging back into the equation for the width we get

\begin{aligned} \frac{2}{||w||} & = 4 \sqrt{2} \\ \frac{2}{\sqrt{2}c} & = 4 \sqrt{2} \\ c = \frac{1}{4} \end{aligned}

Hence the parameters (or coefficients) are in fact $$ w = [\frac{1}{4},-\frac{1}{4}] \ \ b = -\frac{3}{4}$$


(I'm using scikit-learn)

So am I, here's some code to check our manual calculations

from sklearn.svm import SVC
clf = SVC(C = 1e5, kernel = 'linear')
clf.fit(X, y) 
print('w = ',clf.coef_)
print('b = ',clf.intercept_)
print('Indices of support vectors = ', clf.support_)
print('Support vectors = ', clf.support_vectors_)
print('Number of support vectors for each class = ', clf.n_support_)
print('Coefficients of the support vector in the decision function = ', np.abs(clf.dual_coef_))
  • w = [[ 0.25 -0.25]] b = [-0.75]
  • Indices of support vectors = [2 3]
  • Support vectors = [[ 2. 3.] [ 6. -1.]]
  • Number of support vectors for each class = [1 1]
  • Coefficients of the support vector in the decision function = [[0.0625 0.0625]]

Does the sign of the weight have anything to do with class?

Not really, the sign of the weights has to do with the equation of the boundary plane.

Source

https://ai6034.mit.edu/wiki/images/SVM_and_Boosting.pdf

$\endgroup$
4
$\begingroup$

Check this paper on feature selection. The authors use square of weights (of attributes) as assigned by a linear kernel SVM as ranking metric for deciding the relevance of a particular attribute. This is one of the highly cited ways of selecting genes from microarray data.

$\endgroup$
2
$\begingroup$

A great paper by Guyon and Elisseeff (2003). An introduction to variable and feature selection. Journal of machine learning research, 1157-1182 says: "Constructing and selecting subsets of features that are useful to build a good predictor contrasts with the problem of finding or ranking all potentially relevant variables. Selecting the most relevant variables is usually suboptimal for building a predictor, particularly if the variables are redundant. Conversely, a subset of useful variables may exclude many redundant, but relevant, variables."

Therefore I recommend caution when interpreting weights of linear models in general (including logistic regression, linear regression and linear kernel SVM). The SVM weights might compensate if the input data was not normalized. The SVM weight for a specific feature depends also on the other features, especially if the features are correlated. To determine the importance of individual features, feature ranking methods are a better choice.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.