Decreasing importance of trees in XGBoost

Question

On the XGBoost site, it is stated: "XGBoost mostly combines a huge number of regression trees with a small learning rate. In this situation, trees added early are significant and trees added late are unimportant" (https://xgboost.readthedocs.io/en/stable/tutorials/dart.html). AFAIK, the weight applied to each tree's output is the learning rate, which is identical for each tree. When they say that trees added late are unimportant, are they saying that the weights applied to the outputs of these trees are smaller? Or that they're unimportant from an interpretive perspective because they're fitted on the residuals of residuals of residuals of residuals, etc... ?

Answer 1

Your second intuition is correct. The learning rate/shrinkage parameter $\eta$ is generally stable through the ensemble growing process. The latter trees are fitted "on the residuals of residuals of residuals of residuals" of shorts and thus their importance is diminished compared to the earlier trees. As we see in Figure 1 from Rashmi & Gilad-Bachrach's DART: Dropouts meet Multiple Additive Regression Trees the average contribution of a tree added later in the ensemble is quite minor compared to the earlier trees. Having higher shrinkage/lower learning rate $\eta$ does help alleviate this to an extent, but even then the latter trees are not as informative as the ones during the first rounds. In contrast, following the DART procedure, we see that the latter trees still have a significant average contribution as their average prediction remains somewhat on par with that of the earlier trees. I recommend reading the paper for more details, it is well-written and easy to follow.