Collinearity of features and random forest

Question

According to this blog post, inclusion of correlated features in a random forest may be an issue. Which methods do people use in R to detect correlated features and how does one decide which feature to ignore? Could I simply use the variance inflation factor or something similar?

Answer 1

Actually, the blog post does not say that there is an issue with correlated features. It says only that the feature importances that they calculated did not yield correct answer. Now, this does not have to be a problem with random forest itself, but with the feature importance algorithm they used (for example, the default one seems to be biased). They also noticed that including the correlated feature did not hurt the cross-validation performance on the particular dataset they used. So the question is if you want to make predictions, or use the model to infer something about the data?

By design random forest should not be affected by correlated features. First of all, for each tree you usually train on random subset of features, so the correlated features may, or may not be used for a particular tree. Second, consider extreme case where you have some feature duplicated in your dataset (let's call them $A$ and $A'$). Imagine that to make decision, a tree needs to make several splits given this particular feature. So the tree makes first split on $A$, then it may make second split using either $A$ or $A'$. Since they are the same, it could as well throw a coin to choose between them and get exactly the same result. If they are not perfectly correlated, this is only a question if decision tree can pick between features the one that works better, but that's a question about quality of the algorithm in general, not about correlated features.

Answer 2

As pointed out in the comments you should clarify what is your purpose and the data at hands. Several models such as partial least squares take advantage of correlated predictors.

The findCorrelation function in the caret package could serve to detect and remove correlated predictors according to prespecified treshold

descrCor <- cor(filteredDescr)
highlyCorDescr <- findCorrelation(descrCor, cutoff = .75)
filteredDescr <- filteredDescr[,-highlyCorDescr] 

The You should check this source related to preprocessing in Caret

Answer 3

@Tim made some excellent points. One method of determining what (if anything) colinearity is doing to your results is to use the perturb package in R. This adds random noise to the variables and then sees what happens to the results. It requires "some kind of model object" but that need not be a regression. I don't know whether the random forest methods in R produce the right kind of object (I am not a user of random forests).