8
$\begingroup$

I have a random forest binary classifier, but the results from the feature importances are somewhat erratic. Here's what I want to know: Does multicollinearity mess up feature_importances_ in a RandomForestClassifier?

I'm using sci-kit learn (sklearn in python) for the random forest classifier, and getting the feature importances.

To validate the output of feature_importances_ from the RandomForestClassifier in sklearn, I tried removing the most important feature each time (i.e. the feature with the highest feature importance, to see if the second most important feature would appear as the most important feature in the next iteration.... But, this never happened.

The results of this were very erratic, and the order of feature importances was not preserved. At every iteration, there was one feature with extremely high importance (like 0.7 or 0.8), all of the others between 0.1 and 0.0001. There were 9 features to start. The second highest feature importance never appeared as most important in the next iteration.

Does multicollinearity mess up the feature importances, or is there something else I'm missing that messes this up?

$\endgroup$

1 Answer 1

5
$\begingroup$

Yes, multicollinearity definitely can affect variable importances in random forest models. Intuitively, it can be difficult to rank the relative importance of different variables if they have the same or similar underlying effect, which is implied by multicollinearity. That is- if we can access the underlying effect by measuring more than one variable, it's not easy to say which is causing the effect, or if they are mutual symptoms of a third effect.

A discussion of this property of random forests (and of regression questions more generally) can be found in the following lecture notes, among other sources:

http://www-bcf.usc.edu/~shihs/shih_randomforests.pdf

One common way to adjust for this is in the variable selection stage- by selecting one of the multicollinear variables to keep while removing others. This comes, of course, with its own potential issues- by removing potentially partially unique effects.

$\endgroup$
4
  • $\begingroup$ How do I know which of the variables is part of the multicollinearity? Is PCA a way to do this? Also, I guess it isn't clear to me how to determine which one to remove after the multicollinearity has been identified. $\endgroup$
    – makansij
    Jan 5, 2016 at 5:44
  • $\begingroup$ Thanks for the link to the lecture notes. However, it is difficult to interpret them without the lecture, since they seem a bit like disconnected thoughts. Do you have other sources? Thanks. $\endgroup$
    – makansij
    Jan 5, 2016 at 6:15
  • 2
    $\begingroup$ PCA is one way to deal with collinearity- most variable selection methods will reduce collinearity. A brief overview of a few methods is available here: learnitdaily.com/… A more complete treatment of collinearity is available here: biom.uni-freiburg.de/Dateien/PDF/… $\endgroup$ Jan 5, 2016 at 15:34
  • 1
    $\begingroup$ link is broken, but I think it's moved to: www-bcf.usc.edu/~shihs/shih_randomforests.pdf $\endgroup$ Aug 31, 2018 at 3:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.