I'm little bit confused about the integration of using feature selection (SequentialFeatureSelector with k_features="best") with RandomForest or XGboost models

  • When using tree base algorithms like random-forest or xgboost the models are making their own feature selection algorithm (they don't use the whole set of features).

  • When using feature selection approach with wrapper method, the running model use the X features at each step and run algorithm to tests its score (the score of the algorithm when using X features).

  • But if we are using SequentialFeatureSelector with RandomForest or XGboost models, the SFS algorithm runs the model on each step with X features (but behind the scene the model gives the score with less (maybe random set of features)).

  • For example: We can get 10 best features with SFS & XGboost, but after doing some hyper-parameters for xgboost with the dataset of the selected 10 features, we can get winning tree with 6 features.

So is it right using feature selection approach with wrapper method of random forest or xgboost ?


1 Answer 1


Random Forest and XGBoost don't select features. They inherently subsample the features at each tree/boosting iteration, which is part of their procedure to fight with overfitting. So, XGBoost doesn't produce a winning three with, say, 6 features.

The feature selector algorithm can use any estimator, and it selects the features greedily (either forward or backward) based on the scoring function.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.