Permutation feature importance (PFI) is a nice way of getting feature importance in black-box models or models where it is difficult to characterise the relationship between the features and the response. However, it suffers from when the features are highly correlated, which can lead to weird results.

Anyone has experienced with this technique on a correlated dataset? Two solutions I am thinking about are:

  1. Remove the highly colinear altogether.
  2. Group the highly correlated together and shuffle their values at the same time (rather than doing it one column at a time) to get a single importance score for that group of correlated features.

Anyone has other propositions?

  • $\begingroup$ In the spirit of RF you can do bagging, which should remove the correlations. $\endgroup$ Commented Sep 27, 2018 at 6:33
  • $\begingroup$ It might help to clarify what the "weird results" are that you see when doing PFI "the normal way" on datasets with correlated features, and which you're trying to avoid by the altered method. $\endgroup$
    – R.M.
    Commented Oct 1, 2018 at 20:48

1 Answer 1


Regularize. If you regularize your model and inputs, you should remove the issue of multi-collinearity.

Correlation is not causation, however, and so while you can use your method to assign "predictive power" to your features, you cannot establish any sort of causal relationship.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.