Is SVM-RFE a "filter" or a "wrapper" feature selection algorithm? I am wondering whether SVM based recursive feature elimination (Guyon et al., 2000) would be considered a filter or a wrapper method.
On the one hand, it "wraps" around an SVM to get the lowest-weight features.
On the other hand, the trained model is never applied to any test set.
What do you think?
 A: SVM-RFE is a wrapper because it features training a classifier.
Filters use some other criteria (mostly dependence tests) and the only classifier trained is the final one, built on the selected features.
A: SVM-RFE is a embedded feature selection method because it training a classifier to get weights of features to rank them from highest to lower. However, it never gets feedback from classifier like wrapper models do. Therefore, it is a method between filter and wrapper to avoid computational complexity of wrappers and providing multivariate feature ranking in the concept of filter method.
A: RFE is a bit of a hybrid. It looks and acts as a wrapper, similar to backward selection. But its main drawback is its selection of variables is essentially univariate. It uses a univariate measure of goodness to rank order the variables. In this sense it behaves like a filter.
I like this description of feature selection methods:
Filters: fast univariate measure process that relates a single independent variable to the dependent variable. A filter can be linear (e.g., Pearson correlation) or nonlinear (e.g, univariate tree model). Filters typically scale linearly with the number of variables. We want them to be very fast , so we use a filter first to get rid of many candidate variables, then use a wrapper.
Wrappers: called such because there is a model "wrapped" around the feature selection method. A good wrapper is multivariate and will rank order variables in order of multivariate power, thus removing correlations. Wrappers build many, many models and scale nonlinearly with the number of variables. It's best to use a fast, simple nonlinear model for a wrapper (eg, single decision tree with depth about 5, random forest with about 5 simple trees, LGBM with about 20 simple trees). It really doesn't matter what model you use for the wrapper (DT, RF, LGBM, Catboost, SVM...). as long as it's a very simple nonlinear model. After you do the proper wrapper the result is a sorted list of variables in order of multivariate importance. You don't get this with RFE.
In practice you might create thousands of candidate variables. You then do feature selection to get a short list that takes correlations into account. You first do a filter, a univariate measure, to get down to a short list of maybe 50 to 100 candidate variables. Then you run a (proper) wrapper to get you list sorted by multivariate importance, and you typically find maybe 10 to 20 or so variables are sufficient for a good model. Then you use this small number for your model exploration, tuning and selection.
Sure, many nonlinear models by themselves will give you a sorted list of variables by importance, but it's impractical to run a full complex nonlinear model with hundreds or thousands of candidate variables. That's why we do feature selection as a step to reduce the variables before we explore the final nonlinear models.
RFE/RFECV is a poor stepchild in between these. Its variable ranking is essentially univariate, like a filter. It doesn't remove correlations, also like a filter. But it has a model wrapper around it, so it looks like a wrapper. It decides how many variables are ranked #1, sorts the rest by univariate importance, and doesn't sort the #1 variables (or any variables) by multivariate importance.
My opinion: RFE is a popular but lousy wrapper. Use sequentialfeatureselector.
A: It may be a filter method, more precisely, it is Feature Ranking,
Finally, it returns a ranked feature list.
