3
$\begingroup$

I applied libsvm to build a text classifier. The output looks like as follows:

w[a_b] = 0.006
w[a_c] = 0.032
w[a_ctif] = 0.000
w[a_cs] = 1.009
w[aa_d] = 0.000
w[a_e] = 0.001

The terms, e.g., a_b, and a_c, etc are the features generated for characterizing these documents. My question is how to understand these values corresponding to each term? The model output shows that all of these values are bigger than zero. Can I say that a positive file tends to have a term with higher weight?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

SVM weights are calculated according to $\textbf{w}$ = $\sum_{i}$$\alpha_i$$y_i$$\textbf{x}_i$ (good overview here). These give insight into the importance of a feature to the classfier's ability to find the decision boundary between classes. Generally, if a feature has a positive weight, its presence contributes to the separating ability of the classifier, while if negative, it is the absence that is informative (see, for example, Lee et al. 2011).

Also, this question, especially point 3 and the comment about the SVM-RFE algorithm (linked in the question) should be helpful as well.

Improve this answer
$\endgroup$
6
  • $\begingroup$ thanks a lot. By the way, looks like all of the weight values output by libsvm are positive. Is that just because of their specific solver tends to transform every weight value to a positive one? And SVM theory itself should allow the existences of both positive and negative weight values. $\endgroup$
    – user3269
    Mar 19, 2013 at 17:40
  • $\begingroup$ I don't know why all of your weights are positive. It is not inherent to the algorithm, as negative weights, when they exist, are not only possible but also can be equal in magnitude to the most positive. Perhaps your data is such that the presence of all of the features is what allows the decision boundary to be found. $\endgroup$
    – learner
    Mar 19, 2013 at 17:51
  • $\begingroup$ I tried several more data sets, but the result shows that all of the weights are positive. I also feel very confused about that. Searching the SVM-related paper does not return any insight. If a feature's weight value is higher than the value of another feature, can I safe to claim that this feature is more important to building classifier? $\endgroup$
    – user3269
    Mar 19, 2013 at 18:38
  • $\begingroup$ An alternative explanation is that your features are derived from the data set (as opposed to being predetermined, as in Lee et al.), and are all guaranteed to be present. If this is the case, I would think it impossible for you to have negative feature weights. Generally, the greater in magnitude the weight, the more important the feature is to the classifier. You should read the Guyon et al. paper in the linked CV post. $\endgroup$
    – learner
    Mar 19, 2013 at 18:45
  • $\begingroup$ thank you very much for your keeping answering this question. Yes, the features are extracted from the data set.But why the feature weights will always be larger than zero for this case? weight vector should satisfy wx+b =0. And the entries should have some nonzero values to satisfy this equation. I am kind of confusing on this. $\endgroup$
    – user3269
    Mar 19, 2013 at 19:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.