I'm trying to understand the difference between these a bit better. I understand pretty well how random forests work but I guess I'm more hazy on rulefit and how exactly it's different. I know rulefit will incorporate linear components and so can fit linear trends better. What other ways do they differ?
In fact, RuleFit does excessive pruning on a random forest. It tries to find a set of rules generated by random forest to obtain accuracy as close as possible to the accuracy of random forest while reducing the number of rules tremendously. Finally, it builds a model consisting of simple and short rules which are extracted from random forest and builds a comprehensive and understandable model from random forest which is a black box model. How ? It builds a linear model from random forest rules and using an optimization method (Lasso) finds a sparse weight vector that determines which rules are the most important ones. At the end few rules have non-zero weights and the rest of the rules are removed from the ensemble. There are also similar methods with the same aim such as NodeHarvest, but RuleFit has better performance.
They differ in their approaches to tree generation and selection of baselearners to retain for the predictive model:
RuleFit first generates a boosted decision tree ensemble. That is:
- It sequentially grows trees on a pseudo response variable, where the pseudo response for each tree is corrected for the predictions of earlier trees. The amount of correction for earlier trees is controlled by the learning rate (.01, by default).
- For induction of each tree, a random (sub or bootstrap) sample of the training data is selected (same as random forest).
- After generating the boosted decision tree ensemble, all nodes from all trees in the decision tree ensemble are transformed into dummy coded rule variables (taking a value of 0 if the conditions of the rule do not apply, a value of 1 if they do). The final predictive model is selected through sparse regression on the rule variables and the original predictor variables.
A random forest is a decision tree ensemble, in which
- The trees are not induced sequentially, i.e., the response variable is not corrected for predictions of earlier trees.
- For every split in every tree, a random subset of
mtrypredictor variables are selected as potential candidates for the split.
- For induction of each tree, a random (sub or bootstrap) sample of the training data is selected (same as in boosting and RuleFit).
- The final predictive model averages over the predictions of each tree in the ensemble.
Btw, R package pre also supports fitting prediction rule ensembles using a random-forest-type approach.