Is there any academic work on any Decision Tree that fits a regression at its final leaf node?

For instance, suppose I have 100 features (X), and use them build a tree with 3 depths such that I have 8 leaf nodes.

Standard tree approach gives the mean value of Y for the observations in the 8 leaf nodes.

Is there any decision tree algorithm in academic literature that instead does a regression of the Y’s on X for observations in each of the 8 leaf nodes ?

  • $\begingroup$ Yes, as the Wikipedia article mentions, regression trees are a thing. $\endgroup$ – deemel Dec 7 '19 at 16:04
  • 1
    $\begingroup$ There are various algorithms that include regression models in the terminal nodes such as M5, FT, GUIDE, and MOB. See stats.stackexchange.com/questions/141251/… and stats.stackexchange.com/questions/78563/…. In R there is the convenient interface lmtree() in the partykit package (based on MOB), and M5P() in RWeka (approximating M5). $\endgroup$ – Achim Zeileis Dec 7 '19 at 16:30
  • 1
    $\begingroup$ Thanks Achim. That’s exactly what I wanted. Much appreciated $\endgroup$ – uday Dec 7 '19 at 16:38
  • $\begingroup$ @AchimZeileis you can convert it to an answer i guess. $\endgroup$ – gunes Dec 7 '19 at 20:02
  • $\begingroup$ Agree. @AchimZeileis if you make it into an answer I will speech it. Thanks $\endgroup$ – uday Dec 7 '19 at 22:48

Although regression trees with constant fits in the terminal nodes are still much more widely used in practice, there is a long history of literature on regression trees that fit regression models (or other kinds of statistical models) in the nodes of the tree. RECPAM by Ciampi et al. (1988) is pioneering work in the statistical literature, already focusing on survival models in the nodes. M5 by Quinlan (1992) is the first algorithm published in Machine Learning. Other algorithms include GUIDE (Loh 2002), FT (Gama 2004), RD and RA trees (Potts & Sammut 2005), and MOB (Zeileis et al. 2008). An overview of several of these, focusing on MOB in the illustrations, is given by Rusch & Zeileis (2013). The algorithms differ somewhat in the degree to which you have to explicitly specify the regression model considered for the nodes (e.g., as in MOB) vs. using all variable in both the regression and the partitioning (e.g., in M5).

R packages

  • lmtree() in the partykit package (based on MOB)
  • M5P() in the RWeka package (approximating M5)

Previous answers (by @Momo)


  • Ciampi A, Hogg SA, McKinney S, Thiffault J (1988). "RECPAM: A Computer Program for Recursive Partition and Amalgamation for Censored Survival Data and Other Situations Frequently Occurring in Biostatistics. I. Methods and Program Features." Computer Methods and Programs in Biomedicine, 26(3), 239-256.
  • Quinlan R (1992). "Learning with Continuous Classes." In Proceedings of the Australian Joint Conference on Artificial Intelligence, pp. 343–348. World Scientific, Singapore.
  • Loh WY (2002). "Regression Trees with Unbiased Variable Selection and Interaction Detection." Statistica Sinica, 12, 361–386. http://www.jstor.org/stable/24306967
  • Gama J (2004). "Functional Trees." Machine Learning, 55(3), 219–250. doi:10.1023/B:MACH.0000027782.67192.13
  • Potts D, Sammut C (2005). "Incremental Learning of Linear Model Trees." Machine Learning, 61(1-3), 5–48. doi:10.1007/s10994-005-1121-8
  • Zeileis A, Hothorn T, Hornik K (2008). “Model-Based Recursive Partitioning.” Journal of Computational and Graphical Statistics, 17(2), 492–514. doi:10.1198/106186008x319331
  • Rusch T, Zeileis A (2013). "Gaining Insight with Recursive Partitioning of Generalized Linear Models." Journal of Statistical Computation and Simulation, 83(7), 1301–1315. doi:10.1080/00949655.2012.658804
| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.