Permutation feature importance (PFI) is a nice way of getting feature importance in black-box models or models where it is difficult to characterise the relationship between the features and the response. However, it suffers from when the features are highly correlated, which can lead to weird results.

Anyone has experienced with this technique on a correlated dataset? Two solutions I am thinking about are:

  1. Remove the highly colinear altogether.
  2. Group the highly correlated together and shuffle their values at the same time (rather than doing it one column at a time) to get a single importance score for that group of correlated features.

Anyone has other propositions?

  • $\begingroup$ In the spirit of RF you can do bagging, which should remove the correlations. $\endgroup$ – user2974951 Sep 27 '18 at 6:33
  • $\begingroup$ It might help to clarify what the "weird results" are that you see when doing PFI "the normal way" on datasets with correlated features, and which you're trying to avoid by the altered method. $\endgroup$ – R.M. Oct 1 '18 at 20:48

Address the multicollinearity of your data. Unless you're unable to do so without significantly harming the performance of your model, I'm not sure why another approach is needed.

If it's not feasible to mitigate the multicollinearity, I'd look into the methodology of the PFI implementation you're using to see if it can accurately describe the importance of a group of features...you're still running the risk at that point of proposing that features x1, x2, x3, x4, as a group, can explain a certain amount of variance, when in fact only x2 does, or only x3 does, etc. and so you'd have to phrase your findings carefully.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.