At a high level, I don't see how the ensemble of simple models obtained by gradient boosting is better than a single, more complicated model? What's the point of doing gradient boosting instead of a simple more complicated model? Two specific scenarios below:
In an article I read (Gradient boosting from scratch), there is an example of ensemble of simple trees (stumps) for regression. So why is the approach with gradient boosting better than a single more complicated tree of greater depth?
a) In the case of linear regression, seems like it doesn't make sense to use gradient boosting. Can somebody explain why (or rebut)? It would help my understanding of both regression and boosting. For example, instead of doing regression on many features (perhaps even special one, like LASSO), do successive iterations of single-feature regressions, group them via gradient boosting.
b) Same as 2a, only for logistic regression. I suspect here it may make sense, because the calculated function is not linear. But why would one apply gradient boosting with logistic regressions instead of, for example, regularized logistic regression?