I really am at a loss here. I have several tree-based binary classification models trained on a balanced dataset of ~300 samples and ~15000 features. The models have AUROC around 0.8.

I want to retrieve feature importances. I have used the permutation_importance from sklearn, and another function self-made, all return the same values: all zeros except one single feature which has a low value. For other, non tree-based models, everything works fine and feature importances are consistent.

Basically, I have a model that is so robust and redundant that no single feature, when shuffled, is able to change the predictions.

Do I have another option than using the built-in _feature_importances coming from the tree structure, or SHAP values? If I am doing that, how do I compare these with the ones coming from the permutations?


  • 1
    $\begingroup$ What do you mean when you say that for non-tree models the "feature importances are consistent"? Consistent with what? \\ I wonder to what extent the importance results reflect your model having a single predictive feature. One way to investigate this would be to use boruta, which is a feature selection strategy that only retains features which are consistently better than randomized features. $\endgroup$
    – Sycorax
    Commented May 25, 2022 at 15:47
  • $\begingroup$ I mean that some SVM, logistic, etc. models trained on the same dataset, and with similar performance, return values for all features. The values are very similar between the sklearn implementation and my own, they make sense, and they are mostly similar between models (but not always, that is the interesting bit). So I am positive there is nothing wrong with this dataset, which I am working with for more than a year now. It's not just one model, it's all the tree-based ones. It's not just all zeros as a few features have very low values. $\endgroup$
    – SebDL
    Commented May 25, 2022 at 15:59


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.