How to approach feature selection? Let's say I have a dataset with 1000s of features. To save computation resources, I'd like to reduce this number of features.
There seems to be many ways to do feature selection.
Sklearn has some built in features:
http://scikit-learn.org/stable/modules/feature_selection.html
mlxtend also has some interesting algorithms:
https://rasbt.github.io/mlxtend/user_guide/feature_selection/SequentialFeatureSelector
Other than that, I find there isn't much research on best practices.
Ok, don't use RF for feature selection for linear models:
Can a random forest be used for feature selection in multiple linear regression?
For machine learning, XGBoost performs well in Kaggle competitions and in many ways is a good default model.
Is there a good default feature selection technique?
 A: Have a look at Permutation Feature Importance, which is well described here.
Also, have a look here, where this approach is compared to the Random Forest Feature Importance
A: Well regarding feature selection you covered a lot techniques yourself. Another one I would like to add that is used a lot is using LARS or lasso regression for your feature selection. These have the ability of identifying the most important features in your data for regression. Then you could select how many of them you want to keep.
I don't think that there is a default feature selector, but lasso is definitely the most common in kaggle competitions. 
However, I would suggest looking into dimensionality reduction, besides feature selection. For example PCA is an excellent tool for reducing the number of features in your dataset and it accomplishes by creating new features (rather than selecting existing ones).
A: I'd suggest one of two approaches: 
1) Based on the area of application, previous literature have compared many FS techniques in many fields and discussed application processes in their field that are relevant to the type of data. So, the technique is selected based on the most common feature selection techniques in the area.
2) Based on prediction performance, multiple types of FS methods are selected and the technique that show a higher prediction accuracy after training the selected number of features will be chosen.
