how does multicollinearity affect feature importances in random forest classifier?

Question

I have a random forest binary classifier, but the results from the feature importances are somewhat erratic. Here's what I want to know: Does multicollinearity mess up feature_importances_ in a RandomForestClassifier?

I'm using sci-kit learn (sklearn in python) for the random forest classifier, and getting the feature importances.

To validate the output of feature_importances_ from the RandomForestClassifier in sklearn, I tried removing the most important feature each time (i.e. the feature with the highest feature importance, to see if the second most important feature would appear as the most important feature in the next iteration.... But, this never happened.

The results of this were very erratic, and the order of feature importances was not preserved. At every iteration, there was one feature with extremely high importance (like 0.7 or 0.8), all of the others between 0.1 and 0.0001. There were 9 features to start. The second highest feature importance never appeared as most important in the next iteration.

Does multicollinearity mess up the feature importances, or is there something else I'm missing that messes this up?

Answer 1

Yes, multicollinearity definitely can affect variable importances in random forest models. Intuitively, it can be difficult to rank the relative importance of different variables if they have the same or similar underlying effect, which is implied by multicollinearity. That is- if we can access the underlying effect by measuring more than one variable, it's not easy to say which is causing the effect, or if they are mutual symptoms of a third effect.

A discussion of this property of random forests (and of regression questions more generally) can be found in the following lecture notes, among other sources:

http://www-bcf.usc.edu/~shihs/shih_randomforests.pdf

One common way to adjust for this is in the variable selection stage- by selecting one of the multicollinear variables to keep while removing others. This comes, of course, with its own potential issues- by removing potentially partially unique effects.