Let's say I have a dataset with 1000s of features. To save computation resources, I'd like to reduce this number of features.

There seems to be many ways to do feature selection.

Sklearn has some built in features: http://scikit-learn.org/stable/modules/feature_selection.html

mlxtend also has some interesting algorithms: https://rasbt.github.io/mlxtend/user_guide/feature_selection/SequentialFeatureSelector

Other than that, I find there isn't much research on best practices.

Ok, don't use RF for feature selection for linear models: Can a random forest be used for feature selection in multiple linear regression?

For machine learning, XGBoost performs well in Kaggle competitions and in many ways is a good default model.

Is there a good default feature selection technique?


Well regarding feature selection you covered a lot techniques yourself. Another one I would like to add that is used a lot is using LARS or lasso regression for your feature selection. These have the ability of identifying the most important features in your data for regression. Then you could select how many of them you want to keep.

I don't think that there is a default feature selector, but lasso is definitely the most common in kaggle competitions.

However, I would suggest looking into dimensionality reduction, besides feature selection. For example PCA is an excellent tool for reducing the number of features in your dataset and it accomplishes by creating new features (rather than selecting existing ones).

  • $\begingroup$ PCA in general is not suited for preventing overfitting. If you want to select features, do that based on their respective performance. PCA only reduces dimension. Because of this dimensionality reduction, you could lose valuable "information" from your features, while blending in features that actually have a bad performance with features with good performance. Even though using PCA as feature selector/reducer might improve your results, it's not the correct way to go. Just use a proper feature selector or use regularization if you're having problems with overfitting. $\endgroup$ – Anne de Graaf Oct 26 '18 at 21:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.