Lets say that I train a variety of different models - SVM, XGBoost, LogisticRegression, Random Forest, KNeighbors, etc. I then take the permutation importance of each model, and find that Column12 is the most important feature among all the models.

Assuming that this is not a mistake (i.e. hyperparams were tuned and retrained, cross-validation was used, strongly correlated columns were eliminated, etc...), does that mean that Column12 has explicit predictive power as a feature? Why or why not?


1 Answer 1


Does this mean Column12 important to each model? Yes. Does this Column12 mean is important to the outcome (assuming the outcome variable is the same in each model)? Not necessarily.

Permutation importance measures the contribution of altering the feature to the degradation of a model performance metric (typically the mean squared error). Just as its agnostic on the technique that generated the underlying model, it doesn't say anything about importance to the outcome.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.