# Best practices for feature selection?

I have datasets that range from ~2000-9000 columns of predictor variables. I'm often charged with primarily classification - but sometimes regression tasks. I know that I don't need this many variables for my models to be effective but I can't anticipate which ones in a reliable way.

I'm looking for ideas on the general best practices that would cut this down to around 50-150 variables which from my experience, seems fairly effective in determining the outcome.

Currently I'm using lasso or random forest to whittle down the number of variables before running a final model. I want less variables so theres less noise, simply don't need that many, and to make it easier to deploy to production.

• Fewer can be effective if you know without looking at your data which few that represents. Features selection based on relationships with $Y$ can be double dipping and will result in instability and arbitrary selection of variables in many cases. Global penalized regression is often preferred. – Frank Harrell Jul 20 '17 at 22:37
• @FrankHarrell +1 for your second and third sentence but what do you mean with your first sentence? It is slightly more cryptic than thought-provoking... – usεr11852 Aug 5 '17 at 11:21
• I was referring to the illusion of parsimony and misuse of Occam's razor in effect. Fewer is better if the features are pre-specified. If the feature subset is obtained by data torture, all bets are off. As I explain in my course notes from biostat.mc.vanderbilt.edu/rms there is an analogy to Maxwell's Demon where predictive information is "stolen" from the system by attempting to find which predictors to use to predict. – Frank Harrell Aug 5 '17 at 12:20
• Makes sense now, thank you for your clarifications! – usεr11852 Aug 5 '17 at 12:30
• Regarding the idea of using a random forest model first, you might be interested in reading this: Can a random forest be used for feature selection in multiple linear regression? (You would also need to use a randomly selected subset as the training set, & then fit your 'real' model w/ the selected variables on the hold-out set.) – gung - Reinstate Monica Jan 22 '18 at 21:17

Some minor suggestions: I would propose using Elastic Net to potentially have a small amount of $L_2$ regularisation. This should make our feature selection a bit more stable in case of correlated features. Similarly, taking a slightly more sophisticated approach of using Random Forests within a full Recursive Feature Elimination framework like Baruta (See Nilsson et al. for background, CRAN link) instead of simply relying on simple Random Forest variable importance will probably be beneficial.