How is L2 Boosting Different from a Big Regression Tree?

Question

I'm learning about boosting. I think I understand how adaptive boosting works for classification. I'm trying to get some intuition for regression boosting.

At each iteration, adaptive boosting forces the next weak learner to focus more on incorrectly classified points. Intuitively, I can see why that should lead to a good classifier. I could be wrong, but L2 boosting doesn't seem to do anything like that. In L2 boosting, at each iteration, you're fitting a weak learner to the previous iteration's residuals. In a regression tree, when you're splitting a node, aren't you "fitting" the residuals from that node and that node's parents? In both cases, you're "fitting" unweighted residuals, so I don't understand why they're different.

Maybe the the main advantage of L2 boosting over a single tree is that the former has many more regularization/bootstrap-ish options (e.g., randomly choosing subsets of features, the learning rate, the number of trees, individual tree depth, etc.)?

Answer 1

They are different because the set of linear combinations of $N$ regression trees of size $S$ does not include all regression trees of size $ND$. This is easy to see when $S=1$, i.e. decision stumps. In that case, the regression function $f(x)$ obtained by boosting is additive $$f(x) = \sum_{i=1}^p f_i(x_i).$$ If instead one were to grow a regression tree of size $N$, there would potentially be branches including splits of more than one variable. These correspond to interactions and thus lead to a function that does not admit an additive decomposition. This is an important feature of boosted stumps. One can often obtain better performance by boosting with a carefully chosen combination of $N$ and $S$ than by fitting a single regression tree.

Generally, the base learners are chosen with $S$ not very large, e.g. not more than five. On the other hand, random forests are a popular technique that typically uses much larger trees. In that case, bagging is used to reduce the variance associated with these larger trees.