3
$\begingroup$

Let's say I have a dataset with 1000s of features. To save computation resources, I'd like to reduce this number of features.

There seems to be many ways to do feature selection.

Sklearn has some built in features: http://scikit-learn.org/stable/modules/feature_selection.html

mlxtend also has some interesting algorithms: https://rasbt.github.io/mlxtend/user_guide/feature_selection/SequentialFeatureSelector

Other than that, I find there isn't much research on best practices.

Ok, don't use RF for feature selection for linear models: Can a random forest be used for feature selection in multiple linear regression?

For machine learning, XGBoost performs well in Kaggle competitions and in many ways is a good default model.

Is there a good default feature selection technique?

Improve this question
$\endgroup$

3 Answers 3

Reset to default
1
$\begingroup$

Have a look at Permutation Feature Importance, which is well described here.

Also, have a look here, where this approach is compared to the Random Forest Feature Importance

Improve this answer
$\endgroup$
0
0
$\begingroup$

Well regarding feature selection you covered a lot techniques yourself. Another one I would like to add that is used a lot is using LARS or lasso regression for your feature selection. These have the ability of identifying the most important features in your data for regression. Then you could select how many of them you want to keep.

I don't think that there is a default feature selector, but lasso is definitely the most common in kaggle competitions.

However, I would suggest looking into dimensionality reduction, besides feature selection. For example PCA is an excellent tool for reducing the number of features in your dataset and it accomplishes by creating new features (rather than selecting existing ones).

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ PCA in general is not suited for preventing overfitting. If you want to select features, do that based on their respective performance. PCA only reduces dimension. Because of this dimensionality reduction, you could lose valuable "information" from your features, while blending in features that actually have a bad performance with features with good performance. Even though using PCA as feature selector/reducer might improve your results, it's not the correct way to go. Just use a proper feature selector or use regularization if you're having problems with overfitting. $\endgroup$
    – Anne de Graaf
    Commented Oct 26, 2018 at 21:22
0
$\begingroup$

I'd suggest one of two approaches: 1) Based on the area of application, previous literature have compared many FS techniques in many fields and discussed application processes in their field that are relevant to the type of data. So, the technique is selected based on the most common feature selection techniques in the area. 2) Based on prediction performance, multiple types of FS methods are selected and the technique that show a higher prediction accuracy after training the selected number of features will be chosen.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.