# Decision tree that fits a regression at leaf nodes

Is there any academic work on any Decision Tree that fits a regression at its final leaf node?

For instance, suppose I have 100 features (X), and use them build a tree with 3 depths such that I have 8 leaf nodes.

Standard tree approach gives the mean value of Y for the observations in the 8 leaf nodes.

Is there any decision tree algorithm in academic literature that instead does a regression of the Y’s on X for observations in each of the 8 leaf nodes ?

• Yes, as the Wikipedia article mentions, regression trees are a thing. – deemel Dec 7 '19 at 16:04
• There are various algorithms that include regression models in the terminal nodes such as M5, FT, GUIDE, and MOB. See stats.stackexchange.com/questions/141251/… and stats.stackexchange.com/questions/78563/…. In R there is the convenient interface lmtree() in the partykit package (based on MOB), and M5P() in RWeka (approximating M5). – Achim Zeileis Dec 7 '19 at 16:30
• Thanks Achim. That’s exactly what I wanted. Much appreciated – uday Dec 7 '19 at 16:38
• @AchimZeileis you can convert it to an answer i guess. – gunes Dec 7 '19 at 20:02
• Agree. @AchimZeileis if you make it into an answer I will speech it. Thanks – uday Dec 7 '19 at 22:48

Although regression trees with constant fits in the terminal nodes are still much more widely used in practice, there is a long history of literature on regression trees that fit regression models (or other kinds of statistical models) in the nodes of the tree. RECPAM by Ciampi et al. (1988) is pioneering work in the statistical literature, already focusing on survival models in the nodes. M5 by Quinlan (1992) is the first algorithm published in Machine Learning. Other algorithms include GUIDE (Loh 2002), FT (Gama 2004), RD and RA trees (Potts & Sammut 2005), and MOB (Zeileis et al. 2008). An overview of several of these, focusing on MOB in the illustrations, is given by Rusch & Zeileis (2013). The algorithms differ somewhat in the degree to which you have to explicitly specify the regression model considered for the nodes (e.g., as in MOB) vs. using all variable in both the regression and the partitioning (e.g., in M5).

R packages

• lmtree() in the partykit package (based on MOB)
• M5P() in the RWeka package (approximating M5)