34
$\begingroup$

In my understanding, highly correlated variables won't cause multi-collinearity issues in random forest model (Please correct me if I'm wrong). However, on the other way, if I have too many variables containing similar information, will the model weight too much on this set rather than the others?

For example, there are two sets of information (A,B) with the same predictive power. Variable $X_1$,$X_2$,...$X_{1000}$ all contain information A, and only Y contains information B. When random sampling variables, will most of the trees grow on information A, and as a result information B is not fully captured?

$\endgroup$
20
$\begingroup$

That is correct, but therefore in most of those sub-samplings where variable Y was available it would produce the best possible split.

You may try to increase mtry, to make sure this happens more often.

You may try either recursive correlation pruning, that is in turns to remove one of two variables whom together have the highest correlation. A sensible threshold to stop this pruning could be that any pair of correlations(pearson) is lower than $R^2<.7$

You may try recursive variable importance pruning, that is in turns to remove, e.g. 20% with lowest variable importance. Try e.g. rfcv from randomForest package.

You may try some decomposition/aggregation of your redundant variables.

$\endgroup$
  • 3
    $\begingroup$ In some sources, I have seen multicollinearity as having NO effect on the random forest model. For example, here, the most upvoted answer says that "no part of the random forest model is harmed by highly collinear variables". Does this have any validity? $\endgroup$ – Hunle Jan 6 '16 at 6:47
  • 5
    $\begingroup$ I think you're reading the NO too literally. RF models handle quite well correlated/redundant variables, yes. But that does not mean your model necessarily benefits from hoards off unrelated or completely redundant variables(e.g. linear recombinations), it does not crash either. I only advocate modest variable selection, to expect modest improvement of cross-validated model performance. $\endgroup$ – Soren Havelund Welling Jan 6 '16 at 9:23
24
$\begingroup$

Old thread, but I don't agree with a blanket statement that collinearity is not an issue with random forest models. When the dataset has two (or more) correlated features, then from the point of view of the model, any of these correlated features can be used as the predictor, with no concrete preference of one over the others.

However once one of them is used, the importance of others is significantly reduced since effectively the impurity they can remove is already removed by the first feature.

As a consequence, they will have a lower reported importance. This is not an issue when we want to use feature selection to reduce overfitting, since it makes sense to remove features that are mostly duplicated by other features, But when interpreting the data, it can lead to the incorrect conclusion that one of the variables is a strong predictor while the others in the same group are unimportant, while actually they are very close in terms of their relationship with the response variable.

The effect of this phenomenon is somewhat reduced thanks to random selection of features at each node creation, but in general the effect is not removed completely.

The above mostly cribbed from here: Selecting good features

$\endgroup$
  • 3
    $\begingroup$ This has been my go to article for feature selection with RF, as variable importance is often used as metric bmcbioinformatics.biomedcentral.com/articles/10.1186/… Since two years ago I have become more skeptic of feature selection.Feature selection yields overoptimistic cross-validation if not done within a proper outer cross-validation loop. If done properly, I often see none or only little optimization of prediction performance. Now I mainly use feature selection to simplify prediction machines in production or to make a final model more transparent. $\endgroup$ – Soren Havelund Welling Nov 13 '18 at 10:12
  • $\begingroup$ @SorenHavelundWelling - You say that "Feature selection yields overoptimistic cross-validation if not done within a proper outer cross-validation loop". Can you explain that, or refer to a source explaining that? It goes against everything I've read so far... $\endgroup$ – Jack Fleeting Mar 27 '19 at 16:11
  • $\begingroup$ stats.stackexchange.com/questions/27750/… $\endgroup$ – Soren Havelund Welling Mar 28 '19 at 17:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.