(Boosted) regression trees versus model trees - rule of thumb what to use when

Question

I apply (boosted) regression trees to build predicitive models with continuous outcome (xgboost and gbm). While regression trees (rt) split the feature space and predict a constant in each terminal leafe a model tree fits a model in each leafe.

Without too much experience here I would assume that non-boosted trees perform rather bad (too few flexibility) while I know (after some applications) that their boosted versions work well (adding up all those weak learnes splitting with different features and different data we can approximate the target).

I thought about trying out model trees (using glmtree in partykit as here). What can I expect? Do practitioners use model trees? Is the danger of a large generalization error bigger than with regression trees?

Thanks for sharing your experience!

Answer 1

Typically boosted tree-based models, random forests, or other ensemble methods are often superior in terms of predictive accuracy. The biggest advantage of single standalone trees (specifically those with constant fits in each node) is the interpretability. The do not produce black-box results but are easy to understand and communicate to practitioners. Whether there predictive performance is still good enough to justify their use over boosted trees or forests surely depends on the data and the purpose of the analysis.

As for model trees: These try to form a compromise between classical regression models and regression/classification trees with constant fits. They are often still interpretable while improving the predictive accuracy over standard trees. However, they are often still outperformed by boosting/forests.

Model trees have been used in practice but standard constant fit trees as well as boosting/forests are certainly used much more widely.