14
$\begingroup$

I would like to create a random forest using the following process:

  • Build a tree on a random samples of the data and features using information gain to determine splits
  • Terminate a leaf node if it exceeds a pre-defined depth OR any split would result in a leaf count less than a pre-defined minimum
  • Rather than assign a class label for each tree, assign the proportion of classes in the leaf node
  • Stop building trees after a pre-defined number have been constructed

This bucks the traditional random forest process in two ways. One, it uses pruned trees that assign proportions rather than class labels. And two, the stop criteria is a pre-determined number of trees rather than some out-of-bag error estimate.

My question is this:

For the above process that outputs N trees, can I then fit a model using logistic regression with LASSO selection? Does anyone have experience fitting a Random Forest classifier and post-processing with logistic LASSO?

The ISLE framework mentions using LASSO as a post-processing step for regression problems but not classification problems. Furthermore, I don't get any helpful results when googling "Random forest lasso".

Improve this question
$\endgroup$
5
  • $\begingroup$ Lasso is good at finding/weighting useful features when there are many of varying quality. Individual trees in your forest are likely not going to be much better or worse than other trees, so I don't think lasso is going to help you much. $\endgroup$
    – rrenaud
    Mar 12 '13 at 22:43
  • $\begingroup$ By sampling a small fraction without replacement and limiting the tree depth, greater diversity is introduced so i think some form of regularization is warranted. $\endgroup$
    – Zelazny7
    Mar 13 '13 at 22:39
  • $\begingroup$ Can you be more specific about how you plan to fit the logistic model? What exactly are the predictor variables? Also - what is your motivation for post-processing? If you are trying to do variable selection, there are other methods to consider. $\endgroup$
    – ahwillia
    Jan 30 '14 at 13:43
  • $\begingroup$ By outputting the predictions of each tree, a new dataset of predictors is created. This dataset can be used in LASSO regression to arrive at a sparse combination of the tree predictions. The motivation is producing models that are more concise and run more quickly in production. $\endgroup$
    – Zelazny7
    Oct 17 '16 at 15:15
  • $\begingroup$ I encountered similar problems recently, and I found in Friedman's original paper that he designed a loss function specially for binary classification problems. Hope that would be helpful. Besides, do you have any idea on how to extend it to multi-class classification problems? Or what's your approach to multi-class classification problems? $\endgroup$
    – Quan
    Nov 21 '16 at 0:19
5
$\begingroup$

This sounds somewhat like gradient tree boosting. The idea of boosting is to find the best linear combination of a class of models. If we fit a tree to the data, we are trying to find the tree that best explains the outcome variable. If we instead use boosting, we are trying to find the best linear combination of trees.

However, using boosting we are a little more efficient as we don't have a collection of random trees, but we try to build new trees that work on the examples we cannot predict well yet.

For more on this, I'd suggest reading chapter 10 of Elements of Statistical Learning: http://statweb.stanford.edu/~tibs/ElemStatLearn/

While this isn't a complete answer of your question, I hope it helps.

Improve this answer
$\endgroup$
5
  • 3
    $\begingroup$ Thanks. Since I initially posted this question I have become very familiar with R's GBM package. My process now involves building a GBM model of say 10,000 trees and then running all 10,000 trees through GLMnet to perform LASSO regression on the trees. This results in a compressed GBM model with little to no loss in performance (and sometimes a boost). $\endgroup$
    – Zelazny7
    Jan 26 '15 at 14:38
  • $\begingroup$ @Zelazny7 What about on holdout/test data tough Does it predict well? $\endgroup$
    – josh
    Oct 16 '16 at 19:03
  • $\begingroup$ Yes, all of my testing is done on a hold out that does not inform the development in any way. Performance does not degrade in most cases. Sometimes it is a little worse, sometimes it even improves. $\endgroup$
    – Zelazny7
    Oct 17 '16 at 15:13
  • 1
    $\begingroup$ @Zelazny7 I hit upon the same procedure as well (at my last job), with the same experiences. $\endgroup$
    – Matthew Drury
    Nov 21 '16 at 1:54
  • $\begingroup$ You must be on to something...Hastie himself suggests post-processing trees from random forest or boosting using LASSO. He mentioned is in this video at 30:10. $\endgroup$
    – Jonathan
    Sep 30 '19 at 21:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.