I noticed that all feature selection methods implemented in sklearn are based on external estimator that assigns weights to features, AKA feature_importances.

I couldn't find an implementation of the more naive methods, of forward-stepwise selection based on scoring (actually cross-validating the model performance and recording the score) anywhere else but in an external library called mlextend

Is there a reason for that? I realize that scoring has computational cost (you should use CV to get unbiased estimation of the score), but isn't that a better approach as it actually checks performance instead of relying on a heuristic for feature importance?

Will appreciate any insight here.

  • $\begingroup$ What makes feature selection of interest vs. concentrating on predictive accuracy? And you'll probably need penalization to properly pay the price for asking more of the data (what are the "correct" features?) than just prediction. $\endgroup$ – Frank Harrell Feb 18 '16 at 12:39
  • $\begingroup$ As far as stepwise feature selection is concerned, you may look at at RFE together with its RFECV sibling, both from scikit-learn.org/stable/modules/… $\endgroup$ – Sergey Bushmanov Feb 18 '16 at 12:50
  • $\begingroup$ @SergeyBushmanov, RFE and RFECV are exactly the methods based on feature_importances, while I was looking for methods based on scores... $\endgroup$ – ihadanny Feb 18 '16 at 13:07
  • $\begingroup$ @FrankHarrell - my point exactly! I don't understand why is sklearn focusing on the less natural notion of asking the estimator to supply feature_importances instead of just giving predictions, and let a meta-algorithm do the feature selection to reach the best cross-validated prediction score $\endgroup$ – ihadanny Feb 18 '16 at 15:44
  • 1
    $\begingroup$ Feature selection is problematic no matter how it's done. You are spending some of the information in the data to tell you more about the mechanism, instead of concentrating just on prediction. $\endgroup$ – Frank Harrell Feb 19 '16 at 19:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.