RFE vs Backward Elimination - is there a difference?

Question

I recently discovered the RFE tool, and love it. I'd like to understand how this is different from vanilla backward elimination.

Despite lots of information about these two techniques, the penny doesn't seem to drop for me.

Here, the answer intimates that they are essentially the same thing.

Here, the writer suggests that RFE targets individual variable coefficients (I assume p-values or maybe effect size?), whereas Backward Elimination tries to achieve the lowest AIC score for the model as a whole.

Here, the writer suggests that RFE is a type of Backward Elimination, although the explanation is hard to decipher, and the essential difference is not addressed.

So, is RFE just Backward Elimination done by a data scientist, not a statistician?

Answer 1

Quoting Guyon in the paper that introduced RFE:

This [RFE] iterative procedure is an instance of backward feature elimination (Kohavi, 2000 and references therein)

Indeed, when introducing RFE, Guyon does so using Support Vector Machines, and proposes two different methods to rank the single predictors.
At the same time Kohavi tests backward elimination both on tree classifiers and naive bayes - therefore the scoring methods for the features were different.

All and all, the two methods are the same thing - starting from a model with all predictors and removing them one by one based on some scoring function (Z-score for linear regression, Gini for tree based methods, etc.), with the goal of maximizing some target metric (AIC, or test performance).

Answer 2

RFE is a bit of a hybrid. It looks and acts as a wrapper, similar to backward selection. But its main drawback is its selection of variables is essentially univariate. It uses a univariate measure of goodness to rank order the variables. In this sense it behaves like a filter.

I like this description of feature selection methods:

Filters: fast univariate measure process that relates a single independent variable to the dependent variable. A filter can be linear (e.g., Pearson correlation) or nonlinear (e.g, univariate tree model). Filters typically scale linearly with the number of variables. We want them to be very fast , so we use a filter first to get rid of many candidate variables, then use a wrapper.

Wrappers: called such because there is a model "wrapped" around the feature selection method. A good wrapper is multivariate and will rank order variables in order of multivariate power, thus removing correlations. Wrappers build many, many models and scale nonlinearly with the number of variables. It's best to use a fast, simple nonlinear model for a wrapper (eg, single decision tree with depth about 5, random forest with about 5 simple trees, LGBM with about 20 simple trees). It really doesn't matter what model you use for the wrapper (DT, RF, LGBM, Catboost, SVM...). as long as it's a very simple nonlinear model. After you do the proper wrapper the result is a sorted list of variables in order of multivariate importance. You don't get this with RFE.

In practice you might create thousands of candidate variables. You then do feature selection to get a short list that takes correlations into account. You first do a filter, a univariate measure, to get down to a short list of maybe 50 to 100 candidate variables. Then you run a (proper) wrapper to get you list sorted by multivariate importance, and you typically find maybe 10 to 20 or so variables are sufficient for a good model. Then you use this small number for your model exploration, tuning and selection.

Sure, many nonlinear models by themselves will give you a sorted list of variables by importance, but it's impractical to run a full complex nonlinear model with hundreds or thousands of candidate variables. That's why we do feature selection as a step to reduce the variables before we explore the final nonlinear models.

RFE/RFECV is a poor stepchild in between these. Its variable ranking is essentially univariate, like a filter. It doesn't remove correlations, also like a filter. But it has a model wrapper around it, so it looks like a wrapper. It decides how many variables are ranked #1, sorts the rest by univariate importance, and doesn't sort the #1 variables (or any variables) by multivariate importance.

My opinion: RFE is a popular but lousy wrapper. Use sequentialfeatureselector.