1
$\begingroup$

When someone uses a RF, XGBoost or other Decision Tree model for a classification problem, what would you do with the calculated variable importance (that most algorithms provide)?

Would you use it in a way to prune the tree (for data-volume reasons) or if the model performs well you just neglect it?

To me it seems weird to compute the model, use the variable importance options and use the same model/algorithm for your classification model..., how weird are these steps to build a model?

$\endgroup$

3 Answers 3

2
$\begingroup$

To me it seems weird to compute the model, use the variable importance options and use the same model/algorithm for your classification model..., how weird are these steps to build a model?

This is not weird, but exactly what variable importances were developed for! They do not provide a universal measure of relevance of the predictors; they only give a ranking of relevance for each predictor for the fitted model. One could argue it would be weird to use the variable importances of one fitted model to inform the fitting of a different model. You can fit different decision tree ensembles to the same data with (near) identical predictive accuracy, and obtain different values for the variable importances. Furthermore, computation of variable importances (re)uses data points, and the computed importance values will vary, depending on the distribution of these data points, as well as the possible permutation approach used.

Your question seems to imply variable importances should be used to tune the model-fitting parameters of your model, or to perform some kind of inference (i.e., which variables do or do not affect the outcome). They should be used for neither. They quantify the contribution of each variable to the predictions generated by your model. This may be helpful because stakeholders may want to get at least some indication of why a black-box model makes certain predictions, but they obviously do not tell the whole story.

Using variable importances to tune your model will likely lead to overfitting. Using variable importances to perform inference will likely lead to invalid conclusions. E.g., random forests and lasso regression tend to use many more variables for generating predictions, than actually affect the outcome (and, as you point out in one of the comments, xgboost and many random forest implementations suffer from biased variable selection). Predictive accuracy is often not much hurt by this, because of the averaging over many predictor's contributions that the methods perform. But for inference, it can be rather problematic.

$\endgroup$
5
  • $\begingroup$ First of thank you @Marjolein Fokkema for an answer even though the question is so old. I completely agree on a fundamental basis with your first paragraph. My question arises from feature selection tips far and wide spread online, in which variable/feature importance is used to select which of all variables one might want to use in their model, even decision tree models. How would you select variables/feautures for a model? Just those that seem 'logical' in the case you're working on? $\endgroup$
    – xpmatch
    Commented Dec 31, 2021 at 14:32
  • $\begingroup$ As for the second paragraph, I do not fully understand why tuning your model with variable importance will likely lead to overfitting. I assume that tuning a model with variable importance will mean selecting a subset of variables, less variables in the model hence less cause for overfitting. Since overfitting can be due too high dimensional data (vs. e.g. data points). $\endgroup$
    – xpmatch
    Commented Dec 31, 2021 at 14:32
  • $\begingroup$ Furthermore I understand your conclusion as: variable importance should better not be used for inference. Although from other answers and sources I also understand that it is used for decision making. I see a discrepancy in these viewpoints, is that correct or is it more something akin to 'it can, but rather not because of details and if used, be careful'? $\endgroup$
    – xpmatch
    Commented Dec 31, 2021 at 14:32
  • $\begingroup$ @xpmatch In response to: "As for the second paragraph, I do not fully understand why tuning your model with variable importance will likely lead to overfitting..." I could have been more specific: If you would refit the model on the same data, after variable selection using variable importances, I'd expect overfitting. If you would refit the model after variable selection on new data, then not, indeed may even mitigate overfitting. $\endgroup$ Commented Jan 1, 2022 at 12:02
  • $\begingroup$ In response to " 'it can, but rather not because of details and if used, be careful'? " I think that summarizes it well! $\endgroup$ Commented Jan 1, 2022 at 12:03
0
$\begingroup$

If the decision tree models aren't great, the variable importance might suggest to you a feature selection prior to trying another model.

Variable importance can be interesting to the client. It may be that the most "important" variables are those that have most causal influence on the outcome.

However, this is not guaranteed to be the case: an important variable might just be correlated with the outcome, e.g. champagne consumption is predictive of wealth, but not causal.

Note that the second most important variable is the one that is most important, after correlations with the most important are discarded. If you remove from the inputs the most important variable, then some other variable may be promoted. I've even seen some cases where the model's performance was improved by doing this.

$\endgroup$
1
  • 1
    $\begingroup$ When a variable is correlating with the outcome it is not picked up as important, because it does not "add information"? In the last part, aren't you supposed to discard correlated variables before computing the a model and looking up the variable importance? Decision making on which variable of the e.g. pair of correlated variables to discard, is another problem. What were the properties of the variables in your improved model? I read and understand that splitting is biased towards higher cardinality/continuous variables for example. $\endgroup$
    – xpmatch
    Commented Dec 1, 2020 at 10:45
0
$\begingroup$

The only worthwhile application of feature importance measures that I am aware of is to understand what the model has learned at a basic level. This is often necessary to guide decision making. For example, if you want to identify parameters that you can change in order to influence the response, feature importance metrics are useful. I don't think feature selection is generally useful (see, for example, this excellent answer by @gung - Reinstate Monica). At least, it is not useful for improving model performance. I also don't think feature importance measures would be useful for pruning (nor can I imagine why you would be pruning decision trees manually).

Also, note that there are different metrics for 'feature importance' with different interpretations.

$\endgroup$
3
  • $\begingroup$ How do you "id parameters that you can change in order to influence the response" with help of variable importance? Do you somehow imply discarding features? As far as I know feature importance options in e.g. a Random Forest are used for feature selection. $\endgroup$
    – xpmatch
    Commented Dec 1, 2020 at 12:10
  • $\begingroup$ I think that I understand the main points of the poster in your link. Theoretically I agree with the points. Although, what I have witnessed for myself is that results .e.g general performance does seem better with use of feature selection. Overfitting remains a problem when taking into account all variables even for Random Forest. What i mean with performance is train- vs. valid- vs. test-set result are close. Apart from overfitting other reasons for selection could be finding the balance between the speed of the model (training) and performance. $\endgroup$
    – xpmatch
    Commented Dec 1, 2020 at 12:10
  • $\begingroup$ If you have one thousand features which predict whether a process will fail, and you want to prevent the process from failing, feature importance values will identify which features are most important and for which values they have a positive or negative impact on failure. In a real application, you can't necessarily change more than one feature at a time, so it is necessary to use feature importance values to guide your action. $\endgroup$
    – Ryan Volpi
    Commented Dec 1, 2020 at 13:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.