Why do gradient boosting algorithms mostly use trees? Why do gradient boosting algorithms mostly use trees? Is there any logic in this? (in XGBoost and in boosting which in sklearn library uses trees, not other algorithms).
 A: Mostly because they are very good base learner. In few words, I woud say because it is easy to boost trees, and the performance (in terms of predictive power) is very good.
Usually, data mining procedures are well suited for particular applications. For instance, LASSO is a good choice if we believe that the true Data Generating Process (DGP) is sparse, while we do not expect a great perfomance when the DGP is dense. However, there exist several Off-the-shelf procedures, which tend to have discrete to good perfomances in several and various applications, and do not need a lot of data pre-processing. Such procedures satisfy few desiderata: among others, low computational burden, ability to handle mixed data (continuous, binary, categorical, ...) and missing values. Moreover, interpretability is often desired. Black-box models can achieve superior perfomances, but sometimes we do wish understanding what is happening inside.
Decision trees are probably the best off-the-shelf procedure we have so far. However, they have a clear drawback: volatility, which probably is the only thing preventing trees to be the ideal algorithm.
So, boosting trees is considered optimal for two reasons: it is easy, and it solves the problem of high variance (sacrifying interpretability), thus improving accuracy.
This is what comes to my mind. I remember a short discussion about this point in Elements of Statistical Learning (Hastie et al., 2009). Looking at the table of contents, you can probably find more details in chapter 10. I am curious if somebody has other explanations for this.
