lasso regression on top of random forest

Some time ago, I found a paper describing usage of lasso/elastic net regression on binary variables come from random forest. In short, (i,j)-th variable takes 1 if given observation belong to leaf no. i inside tree no. j. Because this will result in a huge number of variables, authors suggested using lasso regression to obtain sparse and interpretable solution.

Does somebody know if such a procedure is implemented in R/Python or can (at least) post a link to this article (or similar). Thanks in advance.

First I think it is hard to say one model out "perform" another. Each model has different pros and cons and should be applied to different cases. For example, I would not say random forest outperforms linear regression, because linear regression is 1. more "stable" 2. requires less computational power 3. more interpretable, plus, if you ground truth between feature and value is really linear, no one can beat linear regression.

Now, back to your question, on code to try two approaches.

You can easily to do the experiment with both way and compare the performance. The trick is using model.matrix in R. Here is one example from ISL book to use model.matrix to convert factors to design matrix and use ridge or lasso.

# Chapter 6 Lab 2 of ISL book: Ridge Regression and the Lasso
library(ISLR)
library(glmnet)
Hitters=na.omit(Hitters)

# transfer formula input to matrix input
x=model.matrix(Salary~.,Hitters)[,-1]
y=Hitters$Salary set.seed(1) train=sample(1:nrow(x), nrow(x)/2) test=(-train) y.test=y[test] grid=10^seq(10,-2,length=100) # The Lasso lasso.mod=glmnet(x[train,],y[train],alpha=1,lambda=grid) plot(lasso.mod) set.seed(1) cv.out=cv.glmnet(x[train,],y[train],alpha=1) plot(cv.out) # get best fit lamda and fit all data bestlam=cv.out$lambda.min
lasso.pred=predict(lasso.mod,s=bestlam,newx=x[test,])
mean((lasso.pred-y.test)^2)
out=glmnet(x,y,alpha=1,lambda=grid)

On the other hand, you can easily do randomForest like

randomforest(Salary~.,data=Hitters)

• How does it relate to my question? May 17 '16 at 16:38
• you were asking if you can do experiment with Python or R, and I listed some code you can try yourself. In addition, I do not believe you can directly compare two types of the model in your way. May 17 '16 at 16:45
• is this related (not randomforest but svm)? please check When to use LIBLINEAR but not LIBSVM May 17 '16 at 16:47
• I asked about procedure (or at least article) which enables to obtain binary variables from randomforest and then applies some regularized regression. I have no interest in comparing randomforest and regression, that's why I considered your post completely irrelevant. May 17 '16 at 18:30
• I was trying to suggest to use model.matrix (the procedure) to "obtain binary variables from random forest then applies some regularized regression". Sorry for your confusion. May 17 '16 at 19:08

This is two years after the original post, but may be of use to others:

R package pre allows for fitting prediction rule ensembles through the algorithm of Friedman & Popescu. It is available from CRAN; development version and an example can be found on GitHub (https://github.com/marjoleinF/pre).

• Actually, I've already used your package! It's really cool. I was wondering, whether you have a plan to add (or maybe it's already done) some survival regreesion rulefit models (like taking the leaf assignments from randomForestSRC for survival analysis task and apply som regularized Cox regression model)? Jun 9 '18 at 13:59
• Cool, thanks! Survival regression is not implemented yet, but should not be too hard to implement. Would be an interesting addition. Will take a look at it this week and let you know when implemented (or when it turns out to be more difficult than I thought). Jun 10 '18 at 10:46
• The current development version on github now supports survival regression. Not extensively tested yet, comments and suggestions are welcome. Here's an example (please excuse lack of line breaks): library("devtools"); install_github("marjoleinF/pre"); library("pre"); data("GBSG2", package = "TH.data"); set.seed(42); surv_pre <- pre(Surv(time, cens) ~ ., data = GBSG2); surv_pre; predict(surv_pre, newdata = GBSG2[1:10,], type = "response"); coef(surv_pre)[1:10,]; imp <- importance(surv_pre); imp Jun 12 '18 at 15:33

This sounds almost like RuleFit described in Friedman & Popescu (2005) in which the lasso is applied to rules formed by an ensemble of trees in addition to linear terms of the variables.

R code is available via the first link.

EDIT: Just noticed that it says on the website that the current RuleFit implementation will not function after Dec 29, 2015 and no updates seem to be available. But give it a try and see if it still works.

• This wasn't article I was talking about, but, anyway, thanks for the link - I 'll definitely try that out. May 17 '16 at 16:41
• Just found this bmcsystbiol.biomedcentral.com/articles/10.1186/… Is that the paper you were talking about? Jun 4 '16 at 10:34