2
$\begingroup$

Why do gradient boosting algorithms mostly use trees? Is there any logic in this? (in XGBoost and in boosting which in sklearn library uses trees, not other algorithms).

$\endgroup$

1 Answer 1

2
$\begingroup$

Mostly because they are very good base learner. In few words, I woud say because it is easy to boost trees, and the performance (in terms of predictive power) is very good.

Usually, data mining procedures are well suited for particular applications. For instance, LASSO is a good choice if we believe that the true Data Generating Process (DGP) is sparse, while we do not expect a great perfomance when the DGP is dense. However, there exist several Off-the-shelf procedures, which tend to have discrete to good perfomances in several and various applications, and do not need a lot of data pre-processing. Such procedures satisfy few desiderata: among others, low computational burden, ability to handle mixed data (continuous, binary, categorical, ...) and missing values. Moreover, interpretability is often desired. Black-box models can achieve superior perfomances, but sometimes we do wish understanding what is happening inside.

Decision trees are probably the best off-the-shelf procedure we have so far. However, they have a clear drawback: volatility, which probably is the only thing preventing trees to be the ideal algorithm.

So, boosting trees is considered optimal for two reasons: it is easy, and it solves the problem of high variance (sacrifying interpretability), thus improving accuracy.

This is what comes to my mind. I remember a short discussion about this point in Elements of Statistical Learning (Hastie et al., 2009). Looking at the table of contents, you can probably find more details in chapter 10. I am curious if somebody has other explanations for this.

$\endgroup$
2
  • 1
    $\begingroup$ Thanks, Could you comment on this? > low computational burden. Aren't trees more difficult to compute than linear models? $\endgroup$
    – SoH
    Feb 5, 2022 at 21:53
  • $\begingroup$ I am not sure about which model takes more computational time. Assuming you mean OLS estimation, my guess is that it depends on the size of the design matrix - big matrices take time to be inverted, even if we rely on some trick (e.g., QR decomposition). On the other hand, fitting a tree is really fast. Of course, trees are greedily grown minimizing a heuristic, with the hope that that is a good approximation to the best tree - the space of possible trees is too large to be explored. In practice, running the rpart() command on R takes really low time. $\endgroup$ Feb 5, 2022 at 22:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.