# Won't highly-correlated variables in random forest distort accuracy and feature-selection?

In my understanding, highly correlated variables won't cause multi-collinearity issues in random forest model (Please correct me if I'm wrong). However, on the other way, if I have too many variables containing similar information, will the model weight too much on this set rather than the others?

For example, there are two sets of information (A,B) with the same predictive power. Variable $X_1$,$X_2$,...$X_{1000}$ all contain information A, and only Y contains information B. When random sampling variables, will most of the trees grow on information A, and as a result information B is not fully captured?

That is correct, but therefore in most of those sub-samplings where variable Y was available it would produce the best possible split.

You may try to increase mtry, to make sure this happens more often.

You may try either recursive correlation pruning, that is in turns to remove one of two variables whom together have the highest correlation. A sensible threshold to stop this pruning could be that any pair of correlations(pearson) is lower than $R^2<.7$

You may try recursive variable importance pruning, that is in turns to remove, e.g. 20% with lowest variable importance. Try e.g. rfcv from randomForest package.

You may try some decomposition/aggregation of your redundant variables.

• In some sources, I have seen multicollinearity as having NO effect on the random forest model. For example, here, the most upvoted answer says that "no part of the random forest model is harmed by highly collinear variables". Does this have any validity? – makansij Jan 6 '16 at 6:47
• I think you're reading the NO too literally. RF models handle quite well correlated/redundant variables, yes. But that does not mean your model necessarily benefits from hoards off unrelated or completely redundant variables(e.g. linear recombinations), it does not crash either. I only advocate modest variable selection, to expect modest improvement of cross-validated model performance. – Soren Havelund Welling Jan 6 '16 at 9:23

Old thread, but I don't agree with a blanket statement that collinearity is not an issue with random forest models. When the dataset has two (or more) correlated features, then from the point of view of the model, any of these correlated features can be used as the predictor, with no concrete preference of one over the others.

However once one of them is used, the importance of others is significantly reduced since effectively the impurity they can remove is already removed by the first feature.

As a consequence, they will have a lower reported importance. This is not an issue when we want to use feature selection to reduce overfitting, since it makes sense to remove features that are mostly duplicated by other features, But when interpreting the data, it can lead to the incorrect conclusion that one of the variables is a strong predictor while the others in the same group are unimportant, while actually they are very close in terms of their relationship with the response variable.

The effect of this phenomenon is somewhat reduced thanks to random selection of features at each node creation, but in general the effect is not removed completely.

The above mostly cribbed from here: Selecting good features

• This has been my go to article for feature selection with RF, as variable importance is often used as metric bmcbioinformatics.biomedcentral.com/articles/10.1186/… Since two years ago I have become more skeptic of feature selection.Feature selection yields overoptimistic cross-validation if not done within a proper outer cross-validation loop. If done properly, I often see none or only little optimization of prediction performance. Now I mainly use feature selection to simplify prediction machines in production or to make a final model more transparent. – Soren Havelund Welling Nov 13 '18 at 10:12
• @SorenHavelundWelling - You say that "Feature selection yields overoptimistic cross-validation if not done within a proper outer cross-validation loop". Can you explain that, or refer to a source explaining that? It goes against everything I've read so far... – Jack Fleeting Mar 27 '19 at 16:11
• stats.stackexchange.com/questions/27750/… – Soren Havelund Welling Mar 28 '19 at 17:23

One thing to add to above explanations: based on the experiments in Genuer et al, 2010:

Robin Genuer, Jean-Michel Poggi, Christine Tuleau-Malot. Variable selection using Random Forests. Pattern Recognition Letters, Elsevier, 2010, 31 (14), pp.2225-2236.

When the number of variables were more than the number of observations p>>n, they added highly-correlated variables with the already-known important variables, one by one in each RF model, and noticed that the magnitude of the importance values of the variables changes (less relative value from the y axis for the already-known important variables) BUT the order of importance of variables remained the same and even the order of the relative values remains pretty similar, and they are still significantly recognisable from noisy variables (less-relevant variables). Also check the table in page 2231 when the number of replications (adding highly-correlated variables with two of the previously-known most important variables) increases, the prediction set for each RF model still shows the most important variable is the already-known most important variable.

for variable selection for interpretation purposes, they construct many (e.g., 50) RF models, they introduce important variables one by one, and the model with lowest OOB error rate is selected for interpretation and variable selection.

for variable selection procedure for prediction purposes, "in each model We perform a sequential variable introduction with testing: a variable is added only if the error gain exceeds a threshold. The idea is that the error decrease must be significantly greater than the average variation obtained by adding noisy variables. "