Meaning of 0-weighted linear weights in SVM

Question

I am working with linear SVM (Using SVMlight) and I'm assisting to a weird phenomenon.

The training algorithm weighted some features 0. Does that means such features are irrelevant for the classification?

Looking at the dataset I found that the vectors containing such features belongs only to a category. And that to me sound like the features are very relevant to classify a new observation.

Moreover I have examples of other features belonging to a single category that have a non zero weight and that confuses me even more.

Is that behavior correct or I am missing something?

Answer 1

Probably the best thing to remember is that SVM training algorithms are trying to find the best classifier to your training data, and they're usually doing that by solving a convex program.

Another way to put it is that they're essentially just doing math -- they can't figure out what's "right" at a higher, more intuitive level. They can't take information into account that isn't encoded in the problem. When dealing with SVM algorithms, the data that you feed them and the labels that you assign are all they get.

What that means with zero classifier coefficients is that there is probably an equivalent or better predictor of your data present in the other features (i.e., it's redundant). It may not be an intuitively obvious choice, but it's a choice that that particular algorithm found. A different algorithm (e.g., LibLinear) may have something different to say.

It's best to judge a trained classifier quantitatively, for example with test set accuracy or a different, testable measure.

Answer 2

Have you normalized the data? If not, and you have features measured on very different scales, then your zero coefficients in the equation of the SVM hyperplane may not be zeros, but merely some small numbers, so that corresponding features still contribute to the classification decision. You can remove these features, rebuild the classifier, and check the accuracy of the new classifier.

Theoretically, if certain coefficients in the equation of the SVM hyperplane are equal to zero, then the corresponding features do not contribute to the classification decision.

If you think that certain features are informative, but they are not used by your classifier, that means that the classifier can get good enough separation using other features, so that other features may also be quite informative.