# Why doesn't test error increase for a high number of boosting iterations? Figure 10.13 of The Elements of Statistical Learning

My question refers to the figure 10.13 of The Elements of Statistical Learning. Test error decreases monotonically with the increase in tree iterations. However, I don't understand why the test error does not raise for the high number of iterations. In the figure below the test error increases for the more complex model.

EDIT: The boosting algorithm combines the output of weak learners (often trees) to produce strong predictions.

$$f_{M}(x) = \sum_{m=1}^{M}f_{m}(x;\theta_{m})$$

where M is a tunable hyperparameter that controls trade off between bias/variance.

In the section 10.14.1 of The Elements of Statistical Learning the gradient boosting model was fit to California Housing data. In the figure 10.13 the average absolute error for training and testing data is decreasing monotonically with the number of boosting iterations (M). This is what authors write:

The test error is seen to decrease monotonically with increasing M, more rapidly during the early stages and then leveling off to being nearly constant as iterations increase. Thus, the choice of a particular value of M is not critical, as long as it is not too small

So, I think authors claim that even for a big number of iterations M the test error will level off. However, shouldn't the test error rise for the very complex model (indicated by a high number of M)?

• What do you mean by "tree iterations"?
– Tim
Oct 16, 2021 at 22:13
• Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
– Community Bot
Oct 16, 2021 at 22:17
• Have you accidentally typed "Test error decreases..." in place of "Training error decreases", and similarly throughout the rest of your question? Oct 16, 2021 at 22:18
• No, I meant test error if. I understand why training error decreases for more complex models. I'll edit and clarify my question. Oct 17, 2021 at 10:48

Note the rest of the paragraph you quoted:

This tends to be the case in many applications. The shrinkage strategy (10.41) tends to eliminate the problem of overfitting, especially for larger data sets.

Shrinkage slows down overfitting, but it does happen. Using basically the same setup as in the example, here's what I get when using sklearn set to many more trees (for whatever reason, the leveling off doesn't happen for me until already a bit further than in ESL):

I thought perhaps a big part of this was also the weak learners themselves: eventually they don't have enough capacity to fit to the gradient very well, and so reaching the actual training minimum isn't easy. I tried it with depth-1 trees, and built a classification problem that incorporates an xor-style data generating process (which depth-1 trees can't see), and the result suggests I was wrong about that:

I suppose actually the xor part makes it very hard to find the real process, and instead the trees eventually get built just on the random patterns?

Colab notebook

• Thanks for the notebook! Oct 26, 2021 at 10:03