Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 15 down vote favorite
7

How to obtain a variable (attribute) importance using SVM?

share|improve this question

2 Answers

up vote 6 down vote accepted

If you use l-1 penalty on the weight vector, it does automatic feature selection as the weights corresponding to irrelevant attributes are automatically set to zero. See this paper. The (absolute) magnitude of each non-zero weights can give an idea about the importance of the corresponding attribute.

Also look at this paper which uses criteria derived from SVMs to guide the attribute selection.

share|improve this answer
1  
Has any one of those algos been implemented in R or other software? – gd047 Aug 29 '10 at 9:26
2  
Yes, take a look at the R penalizedSVM package. Other packages of interest are : penalized, elasticnet, ppls, lars, or more generally: cran.r-project.org/web/views/MachineLearning.html – chl♦ Aug 29 '10 at 9:33
up vote 5 down vote

Isabelle Guyon, André Elisseeff, "An Introduction to Variable and Feature Selection", JMLR, 3(Mar):1157-1182, 2003. http://jmlr.csail.mit.edu/papers/v3/guyon03a.html

is well worth reading, it will give a good overview of approaches and issues. The one thing I would add is that feature selection doesn't necessarily improve predictive performance, and can easily make it worse (beacuse it is easy to over-fit the feature selection criterion). One of the advantages of (especially linear) SVMs is that they work well with large numbers of features (providing you tune the regularisation parameter properly), so there is often no need if you are only interested in prediction.

share|improve this answer
I would even recommend the entire book from I. Guyon and coworkers, j.mp/anblwx. The ESL book from Hastie et al., j.mp/bW3Hr4, provides also interesting discussions around this 'hot' topic. – chl♦ Sep 2 '10 at 18:24
I disagree with your claim; FS is interesting on its own for some explanatory information that it delivers (marker/SNPs selection is an example when it is a main aim of analysis). The feature selection overfit is of course a problem, but there are methods to omit it. – mbq♦ Sep 2 '10 at 18:25
I was making the point that FS doesn't necessarily improve predictive importance, and can make it worse. If finding the informative features is of intrinsic importance, then of course FS should be used, but it may well be that predictive performance is compromised if over-fitting the feature selection criterion occurrs (which happens rather easily). For tasks like micro-array analysis, I would use (bagged) ridge regression for predictions and something like the LASSO for determing the key features (for gaining understanding of the biology). There is no need to do both in the same model. – Dikran Marsupial Sep 3 '10 at 7:28

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.