Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
When someone uses a RF, XGBoost or other Decision Tree model for a classification problem, what would you do with the calculated variable importance (that most algorithms provide)?
Would you use it in a way to prune the tree (for data-volume reasons) or if the model performs well you just neglect it?
To me it seems weird to compute the model, use the variable importance options and use the same model/algorithm for your classification model..., how weird are these steps to build a model?
To me it seems weird to compute the model, use the variable importance options and use the same model/algorithm for your classification model..., how weird are these steps to build a model?
This is not weird, but exactly what variable importances were developed for! They do not provide a universal measure of relevance of the predictors; they only give a ranking of relevance for each predictor for the fitted model. One could argue it would be weird to use the variable importances of one fitted model to inform the fitting of a different model. You can fit different decision tree ensembles to the same data with (near) identical predictive accuracy, and obtain different values for the variable importances. Furthermore, computation of variable importances (re)uses data points, and the computed importance values will vary, depending on the distribution of these data points, as well as the possible permutation approach used.
Your question seems to imply variable importances should be used to tune the model-fitting parameters of your model, or to perform some kind of inference (i.e., which variables do or do not affect the outcome). They should be used for neither. They quantify the contribution of each variable to the predictions generated by your model. This may be helpful because stakeholders may want to get at least some indication of why a black-box model makes certain predictions, but they obviously do not tell the whole story.
Using variable importances to tune your model will likely lead to overfitting. Using variable importances to perform inference will likely lead to invalid conclusions. E.g., random forests and lasso regression tend to use many more variables for generating predictions, than actually affect the outcome (and, as you point out in one of the comments, xgboost and many random forest implementations suffer from biased variable selection). Predictive accuracy is often not much hurt by this, because of the averaging over many predictor's contributions that the methods perform. But for inference, it can be rather problematic.
If the decision tree models aren't great, the variable importance might suggest to you a feature selection prior to trying another model.
Variable importance can be interesting to the client. It may be that the most "important" variables are those that have most causal influence on the outcome.
However, this is not guaranteed to be the case: an important variable might just be correlated with the outcome, e.g. champagne consumption is predictive of wealth, but not causal.
Note that the second most important variable is the one that is most important, after correlations with the most important are discarded. If you remove from the inputs the most important variable, then some other variable may be promoted. I've even seen some cases where the model's performance was improved by doing this.
The only worthwhile application of feature importance measures that I am aware of is to understand what the model has learned at a basic level. This is often necessary to guide decision making. For example, if you want to identify parameters that you can change in order to influence the response, feature importance metrics are useful. I don't think feature selection is generally useful (see, for example, this excellent answer by @gung - Reinstate Monica). At least, it is not useful for improving model performance. I also don't think feature importance measures would be useful for pruning (nor can I imagine why you would be pruning decision trees manually).
Also, note that there are different metrics for 'feature importance' with different interpretations.
