# Collinearity of features and random forest

According to this blog post, inclusion of correlated features in a random forest may be an issue. Which methods do people use in R to detect correlated features and how does one decide which feature to ignore? Could I simply use the variance inflation factor or something similar?

• Variance inflation factor is one way. PCA is another way of removing correlated features. – Demetri Pananos Nov 14 '18 at 20:48
• What problem are you trying to solve? How do correlated features cause a problem for you? The blog post you link to discusses the problem of interpreting feature rankings for correlated features. Is that the problem you are concerned about? Or something else? – Sycorax Nov 14 '18 at 21:10
• The bit: it has issues with correlated data is what I referring to ... – cs0815 Nov 14 '18 at 21:51
• "It has issues" is not a problem that is clearly defined. I don't know what issues you're concerned about or what problem those issues present to you. – Sycorax Nov 14 '18 at 22:27
• Welcome to CV, cs0815. I think @sycorax asked a good question. That is, is the problem you are trying to deal with the one mentioned in the blog post. – Peter Flom Nov 15 '18 at 11:18

Actually, the blog post does not say that there is an issue with correlated features. It says only that the feature importances that they calculated did not yield correct answer. Now, this does not have to be a problem with random forest itself, but with the feature importance algorithm they used (for example, the default one seems to be biased). They also noticed that including the correlated feature did not hurt the cross-validation performance on the particular dataset they used. So the question is if you want to make predictions, or use the model to infer something about the data?

By design random forest should not be affected by correlated features. First of all, for each tree you usually train on random subset of features, so the correlated features may, or may not be used for a particular tree. Second, consider extreme case where you have some feature duplicated in your dataset (let's call them $$A$$ and $$A'$$). Imagine that to make decision, a tree needs to make several splits given this particular feature. So the tree makes first split on $$A$$, then it may make second split using either $$A$$ or $$A'$$. Since they are the same, it could as well throw a coin to choose between them and get exactly the same result. If they are not perfectly correlated, this is only a question if decision tree can pick between features the one that works better, but that's a question about quality of the algorithm in general, not about correlated features.

• Thanks. Before I read this I also thought that this only matters for old regression ... Maybe I missed this ... – cs0815 Nov 14 '18 at 21:55

As pointed out in the comments you should clarify what is your purpose and the data at hands. Several models such as partial least squares take advantage of correlated predictors.

The findCorrelation function in the caret package could serve to detect and remove correlated predictors according to prespecified treshold

descrCor <- cor(filteredDescr)
highlyCorDescr <- findCorrelation(descrCor, cutoff = .75)
filteredDescr <- filteredDescr[,-highlyCorDescr]


The You should check this source related to preprocessing in Caret

@Tim made some excellent points. One method of determining what (if anything) colinearity is doing to your results is to use the perturb package in R. This adds random noise to the variables and then sees what happens to the results. It requires "some kind of model object" but that need not be a regression. I don't know whether the random forest methods in R produce the right kind of object (I am not a user of random forests).