I have a random forest binary classifier, but the results from the feature importances
are somewhat erratic. Here's what I want to know: Does multicollinearity mess up feature_importances_
in a RandomForestClassifier?
I'm using sci-kit learn
(sklearn
in python) for the random forest classifier, and getting the feature importances.
To validate the output of feature_importances_
from the RandomForestClassifier in sklearn, I tried removing the most important feature each time (i.e. the feature with the highest feature importance, to see if the second most important feature would appear as the most important feature in the next iteration.... But, this never happened.
The results of this were very erratic, and the order of feature importances was not preserved. At every iteration, there was one feature with extremely high importance (like 0.7 or 0.8), all of the others between 0.1 and 0.0001. There were 9 features to start. The second highest feature importance never appeared as most important in the next iteration.
Does multicollinearity
mess up the feature importances, or is there something else I'm missing that messes this up?