Using an SVM for feature selection

Lets say I have a highly dimensional classification problem with a lot of noise, and I want to improve my results by removing some of the noisy variables. I've read several papers on using SVMs for feature selection, but I'm at a loss as to how to implement this in R. Are there pre-existing packages that do this, or am I going to have to roll my own?

As I understand them, SVMs have built-in regularization because they tend to penalize large weights of predictors which amounts to favor simpler models. They're often used with recursive feature elimination (in neuroimaging paradigms, at least).

About R specifically, there's the kernlab package, by Alex Smola who co-authored Learning with Kernels (2002, MIT Press), which implements SVM (in addition to e1071). However, if you are after a dedicated framework, I would warmly recommend the caret package.

• thanks for the suggestions. I've looked into the caret package, and it seems that rfe is limited to using linear models, random forests, bagged trees, naive bayes, or ROC curves for feature selection. Is there any kind of variable importance measure that can be computed directly from the SVM? – Zach May 11 '11 at 21:07
• @Zach I thought there was an example of using rfe with SVM. I have to check then. – chl May 12 '11 at 6:42
• @Zach Oups, I forgot to reply. Still, no clear idea wrt. caret package, but it seems that the pathClass has some options for SVM+RFE. Didn't have a chance to test it, though. – chl May 18 '11 at 12:51

For Recursive Feature Extraction (SVM-RFE) the packages e1071 and Kernlab doesn't implement it i think. For the Weka SVMAttributeEval package is for Java i think, but the question was for R as i saw. The best way is trying to implement the SVM-RFE using e1071 and LIBSVM library I found a good parper relating that here.

The Weka SVMAttributeEval package allows you to do feature selection using SVM.

It should be pretty easy to dump your R data frame to a csv file, import that into Weka, do the feature selection, and then pull it back into R.