Can we apply the concept of ridge regression in random forest for predicting the values in order to get more accurate results?

Random forest using regression trees for the prediction. When there is a problem of multicollinearity we will use ridge regression. Multicollinearity definitely can affect variable importances in random forest models. To overcome those multicollinearity in random forest can we use the concept of ridge regression?

  • 2
    $\begingroup$ What do you wish to achieve? Why would using Ridge Regression not be enough? What do you expect from combining both concepts? $\endgroup$ – Nikolas Rieble Jan 18 '17 at 8:51
  • $\begingroup$ To get more accurate results.Random forest using regression trees for the prediction . When there is a problem of multicollinearity we will use ridge regression. multicollinearity definitely can affect variable importances in random forest models. To overcome those multicollinearity in random forest ..can we use the concept of ridge regression $\endgroup$ – omkesh Jan 18 '17 at 9:19
  • 1
    $\begingroup$ I recommend you to read questions that are higly upvoted, therefore you can get a grasp of what is required for a question to recieve attention and good answers. I think your problem has potential and seems relevant, but you will have to elaborate $\endgroup$ – Nikolas Rieble Jan 18 '17 at 9:36
  • $\begingroup$ Ridge regression is in fact OLS regression using a covariate matrix row augmented with a diagonal matrix with the square root of the regularization parameter lambda along the diagonal, stats.stackexchange.com/questions/173132/…. This makes me wonder if it could make sense to do random forest regressions using such a row-augmented covariate matrix to implicitly add a ridge penalty? $\endgroup$ – Tom Wenseleers Dec 14 '20 at 15:27

For predictive accuracy, I would not expect multicollinearity to be a problem for random forests. For variable importances, it is much more likely to be a problem.

Combining random forests and penalized (e.g., ridge) regression can be done with R package pre. This package fits prediction rule ensembles, by first fitting a tree ensemble (bagged, boosted and/or random forest) and then selecting the best nodes through penalized regression (lasso, ridge or elastic net). In the following example, we fit a random forest and prediction rule ensemble on the airquality data. In this dataset, there is a substantial negative correlation (~ -.50) between Temp and Wind and a substantial positive correlation (~.40) between Temp and Month:

airq <- airquality[complete.cases(airquality),]

Now we fit a random forest and a prediction rule ensemble (taking a random forest + ridge regression approach through specification of the mtry and alpha arguments, respectively):

rf <- randomForest(Ozone ~ ., data = airq)
re <- pre(Ozone ~ ., data = airq, mtry = ncol(airq)/3, alpha = 0)

Now we request and plot the variable importances:

rf_imp <- randomForest::importance(rf) 
par(mfrow = c(1,2))
barplot(t(rf_imp), main = "random forest")
pre::importance(re, main = "prediction rule ensemble")

varimps RF vs PRE

We see that the variables Temp, Wind and Solar.R have very similar relative importances in the RF and PRE. The relative importances of Day and Month are lower in the PRE than in the RF.


Interesting question. I think you're right that multicollinearity can be a problem in Random Forests. Let's say you have a variable that is completely correlated with another. Both copies then get elected to build trees with. What happens when they are both available is up to the implementation I guess, both in any case the underlying variable, if it's a reasonably important one, will be in many trees, leading the classification to be overexcited about this variable (overconfident and / or overdependent on this value).

The normal regularization parameters of Random Forests are the number of trees and their complexity. I guess you can apply some kind of penalty for including a variable at all in any iteration, or, in the spirit of ridge regression, a penalty for the square of the number of trees that have included each variable. But then, even if that makes sense, it's probably quite hard to get a quick optimization algorithm for this new global constraint on the classifier.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.