68
$\begingroup$

In my understanding, highly correlated variables won't cause multi-collinearity issues in random forest model (Please correct me if I'm wrong). However, on the other way, if I have too many variables containing similar information, will the model weight too much on this set rather than the others?

For example, there are two sets of information (A,B) with the same predictive power. Variable $X_1$,$X_2$,...$X_{1000}$ all contain information A, and only Y contains information B. When random sampling variables, will most of the trees grow on information A, and as a result information B is not fully captured?

Improve this question
$\endgroup$

4 Answers 4

Reset to default
33
$\begingroup$

That is correct, but therefore in most of those sub-samplings where variable Y was available it would produce the best possible split.

You may try to increase mtry, to make sure this happens more often.

You may try either recursive correlation pruning, that is in turns to remove one of two variables whom together have the highest correlation. A sensible threshold to stop this pruning could be that any pair of correlations(pearson) is lower than $R^2<.7$

You may try recursive variable importance pruning, that is in turns to remove, e.g. 20% with lowest variable importance. Try e.g. rfcv from randomForest package.

You may try some decomposition/aggregation of your redundant variables.

Improve this answer
$\endgroup$
2
  • 6
    $\begingroup$ In some sources, I have seen multicollinearity as having NO effect on the random forest model. For example, here, the most upvoted answer says that "no part of the random forest model is harmed by highly collinear variables". Does this have any validity? $\endgroup$
    – makansij
    Commented Jan 6, 2016 at 6:47
  • 6
    $\begingroup$ I think you're reading the NO too literally. RF models handle quite well correlated/redundant variables, yes. But that does not mean your model necessarily benefits from hoards off unrelated or completely redundant variables(e.g. linear recombinations), it does not crash either. I only advocate modest variable selection, to expect modest improvement of cross-validated model performance. $\endgroup$
    – Soren Havelund Welling
    Commented Jan 6, 2016 at 9:23
55
$\begingroup$

Old thread, but I don't agree with a blanket statement that collinearity is not an issue with random forest models. When the dataset has two (or more) correlated features, then from the point of view of the model, any of these correlated features can be used as the predictor, with no concrete preference of one over the others.

However once one of them is used, the importance of others is significantly reduced since effectively the impurity they can remove is already removed by the first feature.

As a consequence, they will have a lower reported importance. This is not an issue when we want to use feature selection to reduce overfitting, since it makes sense to remove features that are mostly duplicated by other features, But when interpreting the data, it can lead to the incorrect conclusion that one of the variables is a strong predictor while the others in the same group are unimportant, while actually they are very close in terms of their relationship with the response variable.

The effect of this phenomenon is somewhat reduced thanks to random selection of features at each node creation, but in general the effect is not removed completely.

The above mostly cribbed from here: Selecting good features

Improve this answer
$\endgroup$
3
  • 5
    $\begingroup$ This has been my go to article for feature selection with RF, as variable importance is often used as metric bmcbioinformatics.biomedcentral.com/articles/10.1186/… Since two years ago I have become more skeptic of feature selection.Feature selection yields overoptimistic cross-validation if not done within a proper outer cross-validation loop. If done properly, I often see none or only little optimization of prediction performance. Now I mainly use feature selection to simplify prediction machines in production or to make a final model more transparent. $\endgroup$
    – Soren Havelund Welling
    Commented Nov 13, 2018 at 10:12
  • $\begingroup$ @SorenHavelundWelling - You say that "Feature selection yields overoptimistic cross-validation if not done within a proper outer cross-validation loop". Can you explain that, or refer to a source explaining that? It goes against everything I've read so far... $\endgroup$
    – Jack Fleeting
    Commented Mar 27, 2019 at 16:11
  • $\begingroup$ stats.stackexchange.com/questions/27750/… $\endgroup$
    – Soren Havelund Welling
    Commented Mar 28, 2019 at 17:23
2
$\begingroup$

One thing to add to above explanations: based on the experiments in Genuer et al, 2010:

Robin Genuer, Jean-Michel Poggi, Christine Tuleau-Malot. Variable selection using Random Forests. Pattern Recognition Letters, Elsevier, 2010, 31 (14), pp.2225-2236.

When the number of variables were more than the number of observations p>>n, they added highly-correlated variables with the already-known important variables, one by one in each RF model, and noticed that the magnitude of the importance values of the variables changes (less relative value from the y axis for the already-known important variables) BUT the order of importance of variables remained the same and even the order of the relative values remains pretty similar, and they are still significantly recognisable from noisy variables (less-relevant variables). Also check the table in page 2231 when the number of replications (adding highly-correlated variables with two of the previously-known most important variables) increases, the prediction set for each RF model still shows the most important variable is the already-known most important variable.

for variable selection for interpretation purposes, they construct many (e.g., 50) RF models, they introduce important variables one by one, and the model with lowest OOB error rate is selected for interpretation and variable selection.

for variable selection procedure for prediction purposes, "in each model We perform a sequential variable introduction with testing: a variable is added only if the error gain exceeds a threshold. The idea is that the error decrease must be significantly greater than the average variation obtained by adding noisy variables. "

Improve this answer
$\endgroup$
-1
$\begingroup$

Let us first correct the notion and widely belief of "highly correlated variables cause multi-collinearity". ive seen countless internet tutorials suggestion to remove correlated variables. First correlation and multicollinearity are two different phenomenon. Therefore, there are instances where there is high correlation but no multi-collinearity, and vice-versa (there is multi-collinearity but almost no correlation). there are even different statistical methods to detect those two.

So to answer your question following this logic, the notion that correlation implies multi-collinearity is incorrect, hence does not necessarily will cause multi-collinearity. and you should use proper statistical methods to detect those two individually.

Improve this answer
$\endgroup$
1
  • $\begingroup$ This reads like an answer to a different question. It doesn’t mention random forest or feature selection. $\endgroup$
    – Sycorax ♦
    Commented Feb 26 at 19:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.