how to decide the best way to calculate feature importance? I got a dataset containing over 1000 features. I tried to find out the most important ones then apply them into model training. It seems I can achieve this by lots of methods, e.g., by examining confidence of linear regression, or by calculating importance directly in XGB classifier. My question is how to decide the best estimator to achieve this? 
 A: The answer provided by tomka for using $L_1$ regularization (LASSO) is correct, although it does have some limitations.


*

*In a setting where you have more predictors than samples (called the '$p\gg N$' problem) $L_1$ regularization is able to have at most $N$ non-zero coeffcients in the model before it saturates.

*$L_1$ regularization is not able to utilize correlation  between coefficients, the $L_1$ regularization tends to randomly select only one  feature from a correlated group.


I would suggest to instead use the elastic net method for this. Elastic net is a hybrid method that incorporates both $L_1$ and $L_2$ regularization. By incorporating $L_2$ regularization (ridge regression) as well, the elastic net allows the $L_2$-tendency of shrinking coefficients for correlated predictors towards each other, while retaining the feature selection provided by the LASSO. This encourages a 'grouping effect', providing subsets of correlated predictors. 
Also the elastic net is able to cope with the limitations of the $L_1$ regularization of the '$p \gg N$' problem.
Furthermore, I assume you are interested in finding statistical significant predictors. Therefore I would also suggest to perform feature selection via the elastic net method in Stability Selection setting.  Basically this comes down to building the model a multitude of times with a randomized subset of the original training data, while observing how often each weight for a variable is not shrunk to zero (i.e. it contributes to the prediction). If a weight variable is stable (e.g not shrunk to zero over 1000 randomized training runs), this may indicate significance.
Sources:


*

*Paper for elastic net: Regularization and variable selection via the elastic net

*Paper on L1 regularization: Regression shrinkage and selection via lasso

*My Msc thesis, where I faced a similar problem


Implementations:


*

*R implementation: CRAN - Package elasticnet

*Python implementation: sklearn.linear_model.ElasticNet
A: If you have a lot of predictors, a useful way to identify the strongest predictors is penalized regression, in particular the LASSO. The advantage of LASSO over other penalized techniques such as Ridge is that it will knock out the weak predictors in the model fully leaving only the strongest predictors in the model, the higher the penalization term becomes.
Besides penalized regression other model selection techniques such as backward or forward inclusion of features based on AIC or BIC criterions may be considered. However since you have a lot of teatures these procedures are prone to finding local solutions (not optimal in terms of minimzed mean squared error).
