Running SequentialFeatureSelector with tree base model seems to be wrong

Question

I'm little bit confused about the integration of using feature selection (SequentialFeatureSelector with k_features="best") with RandomForest or XGboost models

• When using tree base algorithms like random-forest or xgboost the models are making their own feature selection algorithm (they don't use the whole set of features).

• When using feature selection approach with wrapper method, the running model use the X features at each step and run algorithm to tests its score (the score of the algorithm when using X features).

• But if we are using SequentialFeatureSelector with RandomForest or XGboost models, the SFS algorithm runs the model on each step with X features (but behind the scene the model gives the score with less (maybe random set of features)).

• For example: We can get 10 best features with SFS & XGboost, but after doing some hyper-parameters for xgboost with the dataset of the selected 10 features, we can get winning tree with 6 features.

So is it right using feature selection approach with wrapper method of random forest or xgboost ?

Answer 1

Random Forest and XGBoost don't select features. They inherently subsample the features at each tree/boosting iteration, which is part of their procedure to fight with overfitting. So, XGBoost doesn't produce a winning three with, say, 6 features.

The feature selector algorithm can use any estimator, and it selects the features greedily (either forward or backward) based on the scoring function.