How to obtain a variable (attribute) importance using SVM?


If you use l-1 penalty on the weight vector, it does automatic feature selection as the weights corresponding to irrelevant attributes are automatically set to zero. See this paper. The (absolute) magnitude of each non-zero weights can give an idea about the importance of the corresponding attribute.

Also look at this paper which uses criteria derived from SVMs to guide the attribute selection.

  • 1
    $\begingroup$ Has any one of those algos been implemented in R or other software? $\endgroup$ – George Dontas Aug 29 '10 at 9:26
  • 5
    $\begingroup$ Yes, take a look at the R penalizedSVM package. Other packages of interest are : penalized, elasticnet, ppls, lars, or more generally: cran.r-project.org/web/views/MachineLearning.html $\endgroup$ – chl Aug 29 '10 at 9:33

Isabelle Guyon, André Elisseeff, "An Introduction to Variable and Feature Selection", JMLR, 3(Mar):1157-1182, 2003. http://jmlr.csail.mit.edu/papers/v3/guyon03a.html

is well worth reading, it will give a good overview of approaches and issues. The one thing I would add is that feature selection doesn't necessarily improve predictive performance, and can easily make it worse (beacuse it is easy to over-fit the feature selection criterion). One of the advantages of (especially linear) SVMs is that they work well with large numbers of features (providing you tune the regularisation parameter properly), so there is often no need if you are only interested in prediction.

  • 2
    $\begingroup$ I would even recommend the entire book from I. Guyon and coworkers, j.mp/anblwx. The ESL book from Hastie et al., j.mp/bW3Hr4, provides also interesting discussions around this 'hot' topic. $\endgroup$ – chl Sep 2 '10 at 18:24
  • $\begingroup$ I disagree with your claim; FS is interesting on its own for some explanatory information that it delivers (marker/SNPs selection is an example when it is a main aim of analysis). The feature selection overfit is of course a problem, but there are methods to omit it. $\endgroup$ – user88 Sep 2 '10 at 18:25
  • $\begingroup$ I was making the point that FS doesn't necessarily improve predictive importance, and can make it worse. If finding the informative features is of intrinsic importance, then of course FS should be used, but it may well be that predictive performance is compromised if over-fitting the feature selection criterion occurrs (which happens rather easily). For tasks like micro-array analysis, I would use (bagged) ridge regression for predictions and something like the LASSO for determing the key features (for gaining understanding of the biology). There is no need to do both in the same model. $\endgroup$ – Dikran Marsupial Sep 3 '10 at 7:28

If you use R, the variable importance can be calculated with Importance method in rminer package. This is my sample code:

M <- fit(y~., data=train, model="svm", kpar=list(sigma=0.10), C=2)
svm.imp <- Importance(M, data=train)

In detail, refer to the following link https://cran.r-project.org/web/packages/rminer/rminer.pdf

  • 2
    $\begingroup$ This answer is incomplete. It does not describe what the variable importance in that package is trying to communicate. $\endgroup$ – Matthew Drury Sep 14 '17 at 15:05
  • $\begingroup$ I've added sample code $\endgroup$ – Takashi Kaneda Sep 14 '17 at 15:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy