How to rank colinear features

Question

I'm currently working on a paper where I'm training a bunch of random forest classifiers on a feature set with fairly high degree of multicollinearity. In the end I'm aiming to provide an overview, ranking all my features employed by how "important" they are for the overall classification result. Initially I was planning on feature_importances_ attribute of the sklearn random forest implementation.

As described in this question, however, I figured I can't use this method to rank the features due to the high degree of multicollinearity. My question is, if there is any other way of ranking features for classification or regression that is insensitive to multicollinearity?

Answer 1

You can maybe use feature selection methods try to determine which features (or rather, combinations of features) perform the best, as ranked by cross-validation. Common methods are Recursive Feature Elimination, RFC or RFECV in sklearn.

You can also attempt adding features one at a time using Sequential Forward Selection. There is an implementation in mlxtend. How that causes scores to improve as a feature is added can maybe say something about the strength/importance of that feature.