It seems to me that despite the huge variety and development of ML methods (I'm specifically interested in regression methods), OLS is still considered, and often cited, as a benchmark - which makes sense at least because of its easy interpretation.

Despite this, I failed to find regression methods that can be thought to generalize a multivariate OLS, in the sense of being able to identify linear relations between features. I find this surprising because in many real world examples (I work with economic data) linear relations coexist with nonlinearities, and because... in principle, it does not seem like a difficult task.

For instance, the value predicted by a regression tree for an observation only depends on the leaf it ends into. Why not fit a linear regression on each leaf instead, and use the resulting (leaf-specific) coefficients for the prediction? Yes, the cost would (if I understand correctly) dominate the cost of building the tree... but on the other hand, it would still be asymptotically equivalent to the cost of an OLS (with a lower memory footprint). And as long as the tree is not too deep (that is: each leaf still contains enough observations), this would for instance perfectly predict a pure linear relation. It would be a sort of multi-dimensional automatic piecewise linear regression.

Or vice-versa, it would be tempting to run a regression tree on OLS residuals. If done by using wisely the (example from sklearn) min_impurity_decrease argument, this would, it seems to me, reliably dominate OLS in terms of explanatory power, at least for large samples (with the two coinciding in the case of a perfectly linear relation).

Is there a reason why techniques of this kind are not widespread? Or are they?


You may want to consider reading this paper https://arxiv.org/pdf/1510.04342.pdf.

The authors attempt to fit trees to grow the forest and compute an estimate for each leaf to determine the treatment effect on a subpopulation of observations within each leaf. Each leaf is a subpopulation of observations as identified by the trees on some defined split measure typically chosen based on the domain knowledge.

| cite | improve this answer | |
  • $\begingroup$ +1 for the link to the work of Wager and Athey. There has been a number of earlier efforts for trees, though, combining (generalized) linear models with recursive partitioning. For some of these convenient software is also available, e.g., M5P, GUIDE, or MOB. See my comment above. $\endgroup$ – Achim Zeileis Mar 16 '18 at 13:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.