Why Feature Selection with sklearn.feature_selection.SequentialFeatureSelector is a preprocessing task?

Question

I am facing a feature selection problem. Because I am building an Explanatory Regression Model I decided to follow a Forward Sequential Feature Selection.

Moreover I wanted to implement sklearn.feature_selection.SequentialFeatureSelector for features selection.

After reading sklearn documentation about this transformer some doubts raised.

Quoting the documentation :

Feature selection is usually used as a pre-processing step before doing the actual learning. The recommended way to do this in scikit-learn is to use a Pipeline[...]

Why SequentialFeatureSelector is used in pre-processing? Specifically inside a Pipeline?

I supposed to imagine this as an iterative process evaluating the performance metric and computing a statistical significance test in a for-loop, training each time my model with a different number of variables and then evaluating how it performs.

The only reason that I can find for choosing to do forward feature selection before training is for a "computational saving".

It is correct as an assumption and there are other reasons?

Answer 1

I am not sure how much below explanation is relevant to this question but let me try.

Feature selection is the process of including the significant features in the model. We have many options to do but generally we can use below method to reduce number of features in the model.

1. Forward selection method
2. Backward elimination method
3. Stepwise method
4. Recursive feature elimination (RFE)

Internally all above methods are building model and based on rules eliminating insignificant features. Sometimes we use selected significant features in the model as next step. For more detail you can refer below link

http://rasbt.github.io/mlxtend/user_guide/feature_selection/SequentialFeatureSelector/

Answer 2

To see the forest for the trees, i.e., step back and look at the big picture, there are several types of feature selection methods that have been developed specifically for regression - all of which are performed during the regression procedure. Sometimes these approaches are called "fishing expeditions," since you are looking for good features that help model performance. In regression, examples are forward selection, reverse elimination, etc. I usually use reverse to "preserve subset correlation." That is, if you select the best fish from the top of the bucket, you may break up correlation they have with other fish (they are holding hands, but you rip them apart). Using the reverse elimination method will preserve sets of variables which are both correlated and help explain the model variance.

In machine learning, there are numerous other filtering and wrapping approaches for feature selection, like best ranked N, recursive feature elimination, greedy plus-take away, sequential forward-reverse floating methods.

Your question deals with the potential for selection bias, in which the features selected can bias the results because only features which help the model are used. Wrapping methods select features during the discrimination or training (learning) procedure, and may induce more bias than filtering methods, which select features independently or before modeling.

For classification analysis, filtering methods can be as simple as running hypothesis tests involving t-tests, ANOVA, Mann-Whitney, Kruskal-Wallis to identify features that are predictive of (best discriminate) class.

Whereas for regression, filtering can involve correlation analysis before regression to identify features which correlate best with outcome. Correlative features can then be employed during regression analysis (assuming some type of normality obviously or "central tendency").

Overall, I commonly use filtering and avoid wrapping methods, but do on occasion use reverse elimination selection during regression (linear, logistic). For classification, I usually use greedy plus-take away methods for filtering in order to accomplish feature selection, and then train/test classifiers which never "touched" the feature selection methods.