How to handle correlated variables before using Recursive Feature Elimination?

Question

I have seen a few Kaggle notebooks that list without reason that RFE works better when removing correlated variables. I struggle to see the reason why so I conducted some of my own research and would like to verify if my conclusions are correct.

From my research with sklearn's CART algorithm, I have taken a good predictive feature (Feature A) and a highly correlated feature with some extra noise (Feature B). It seems that due to high correlation, their mean impurity decrease is very similar and the splits roughly split the feature importance between the two variables. This creates situations where Feature A and Feature B can be ranked highly if Feature A is a good predictor despite Feature B being redundant. Feature B will likely not be removed for several iterations and reduce the model score for these first few iterations, thereby limiting the combination of features that RFE considers.

But I presume there are multiple factors that influence the way we identify a "highly correlated variable". At which correlation cutoff point do we determine that the variables will hurt the RFE feature selection process?

For example, the number of splits in the algorithm will be one determining factor. If Feature A only has a single split, then Feature B will not benefit from having high correlation with Feature A, and likely be removed by RFE without problems.

Answer 1

Tree classifiers typically use features independently (univariate) for node splits and are not really multivariate. Thus, they're not likely to knock down the importance score of a feature because it's correlated with other features.

I use importance scores to identify features for input into other classifiers with their own feature selection methods, which can reduce the impact of correlated features. This is mostly because tree classifiers were not developed to throw out correlated features.

One of the advantages of random forests (RF), another tree classifier, is that frequencies for all splits don't always track with importance scores. (1st node splits seems to track with importance scores). Below are example importance scores and split frequencies for a dataset showing that the sort order for importance is not the same as the number all node splits. (Split frequencies here are based on descending Gini index for the tree).