How to obtain a variable (attribute) importance using SVM?
3 Answers
$\begingroup$
$\endgroup$
3
If you use l-1 penalty on the weight vector, it does automatic feature selection as the weights corresponding to irrelevant attributes are automatically set to zero. See this paper. The (absolute) magnitude of each non-zero weights can give an idea about the importance of the corresponding attribute.
Also look at this paper which uses criteria derived from SVMs to guide the attribute selection.
-
2$\begingroup$ Has any one of those algos been implemented in R or other software? $\endgroup$– YorgosCommented Aug 29, 2010 at 9:26
-
6$\begingroup$ Yes, take a look at the R penalizedSVM package. Other packages of interest are : penalized, elasticnet, ppls, lars, or more generally: cran.r-project.org/web/views/MachineLearning.html $\endgroup$– chlCommented Aug 29, 2010 at 9:33
-
1$\begingroup$ The website linked to the first "this paper" is no longer valid $\endgroup$ Commented Mar 30, 2021 at 8:59
$\begingroup$
$\endgroup$
3
Isabelle Guyon, André Elisseeff, "An Introduction to Variable and Feature Selection", JMLR, 3(Mar):1157-1182, 2003. http://jmlr.csail.mit.edu/papers/v3/guyon03a.html
is well worth reading, it will give a good overview of approaches and issues. The one thing I would add is that feature selection doesn't necessarily improve predictive performance, and can easily make it worse (beacuse it is easy to over-fit the feature selection criterion). One of the advantages of (especially linear) SVMs is that they work well with large numbers of features (providing you tune the regularisation parameter properly), so there is often no need if you are only interested in prediction.
answered Sep 2, 2010 at 18:07
-
2$\begingroup$ I would even recommend the entire book from I. Guyon and coworkers, j.mp/anblwx. The ESL book from Hastie et al., j.mp/bW3Hr4, provides also interesting discussions around this 'hot' topic. $\endgroup$– chlCommented Sep 2, 2010 at 18:24
-
$\begingroup$ I disagree with your claim; FS is interesting on its own for some explanatory information that it delivers (marker/SNPs selection is an example when it is a main aim of analysis). The feature selection overfit is of course a problem, but there are methods to omit it. $\endgroup$– user88Commented Sep 2, 2010 at 18:25
-
$\begingroup$ I was making the point that FS doesn't necessarily improve predictive importance, and can make it worse. If finding the informative features is of intrinsic importance, then of course FS should be used, but it may well be that predictive performance is compromised if over-fitting the feature selection criterion occurrs (which happens rather easily). For tasks like micro-array analysis, I would use (bagged) ridge regression for predictions and something like the LASSO for determing the key features (for gaining understanding of the biology). There is no need to do both in the same model. $\endgroup$ Commented Sep 3, 2010 at 7:28
$\begingroup$
$\endgroup$
2
If you use R, the variable importance can be calculated with Importance method in rminer package. This is my sample code:
library(rminer)
M <- fit(y~., data=train, model="svm", kpar=list(sigma=0.10), C=2)
svm.imp <- Importance(M, data=train)
In detail, refer to the following link https://cran.r-project.org/web/packages/rminer/rminer.pdf
-
2$\begingroup$ This answer is incomplete. It does not describe what the variable importance in that package is trying to communicate. $\endgroup$ Commented Sep 14, 2017 at 15:05
-