3
$\begingroup$

In this scenario, the end goal is to determine which columns are most important. I use this term very very very loosely, because I know that interpretability is never straightforward but I just want to get a general idea about a ranking for column importance.

Say that I have 5+ different tree-based models that I train on the data. I then gather the percent importance for each of the features, for each of the models. At this point I have a dataset of however many features I started off with, multiplied by however many models I am using, and each of the features' percent importances per model.

I then calculate how many times each feature appear in the top 10 by percent importance, per model. So at this point, I have calculated something like

  • column1 is in the top 10 by percent importance for 8/10 models
    • I can conclude that since column1 is relatively important for a lot of the model, it is a generally important feature
  • column2 is in the top 10 by percent importance for only 2/10 models, and column3 is never in the top 10 for any models
    • I can conclude that column2 and column3 are relatively unimportant

Does this approach make sense/is my interpretation correct?

Notes:

  • I am only using tree-based models which have percent importance attributes to them (RandomForest, CatBoost, ExtraTrees, etc...)
$\endgroup$

2 Answers 2

5
$\begingroup$

I would be careful with this method. Usually, the importance attribute given by a model only refers to this model, i.e. it is just saying that, for this particular model, this column has this special importance. But that doesn't necessarily mean that it is as important in other models.

So the real question is rather about how you want to use the importance. For the application of e.g. Random Forest, you should use only the importance given by Random Forest.

$\endgroup$
1
  • $\begingroup$ My end goal is solely explanability - I won't be making any predictions. I simply want to understand a complex dataset, and communicate which factors are most important to the target, to the end user $\endgroup$ Commented Jul 23, 2022 at 9:43
2
$\begingroup$

In contrast to frank, I think this method is potentially useful. I think we can reasonably disagree here. I am less concerned with what frank sees as a problem: since you already look at multiple models, you are already trying to account for the fact that variable importance is (indeed) only measured within the context of a specific model.

What I would be a bit careful about is any correlation between models. If your models are highly correlated (say, different Bayesian models with the same predictors and only small differences in priors), then they will likely output the same predictors as "important" - but this agreement does not mean much.

So: if you do this, try to ensure your models are as dissimilar as possible. (In which case the question arises whether the importance percentage measures are still sufficiently comparable.)

$\endgroup$
3
  • $\begingroup$ The models that I'm currently using are: randomforest, xgboost, adaboost, extratrees, gradientboostingclassifier, and catboost (xg and cat are the only ones not from sklearn). My high level impression is that these are all "correlated", since they are all build from decision trees. Is this the same thing that you are talking about? $\endgroup$ Commented Jul 23, 2022 at 9:46
  • 1
    $\begingroup$ Yes, exactly. It's hard to quantify the correlation between models. You could also add something like a glmnet model to the mix, that is quite different (but then you run into the variable importance comparability issue). $\endgroup$ Commented Jul 23, 2022 at 9:56
  • $\begingroup$ Would I have to calculate the permutation importance of a different model instead, since k-neighbors for example doesn't have a built-in feature_importance_ attribute? $\endgroup$ Commented Jul 23, 2022 at 10:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.