# Is a high learning rate irrelevant when dropping the first or last tree in a GBDT with 100 trees?

Suppose we've trained a GBDT model with 100 trees with a fairly high learning rate. Consider two cases:

1. We drop the first tree in the model

2. We drop the last tree in the model

We then compare models performance on the train set.

I wrote that in case 1 performance will drop more than in case 2 b/c a high learning rate means less corrections are made with each new tree so presumably the last tree did not contribute that many corrections to the prediction.

The answer in the book is: In the case1 performance will drop more than in the case2. In GBDT model we have sequence of trees, each improve predictions of all previous. So, if we drop first tree — sum of all the rest trees will be biased and overall performance should drop. If we drop the last tree -- sum of all previous tree won't be affected, so performance will change insignificantly (in case we have enough trees).

My question is, is my explanation wrong? Is the learning rate irrelevant in this case? Also, the question italicizes "train set," why is this distinction important?

• This is a fun little question! (+1) Please see my answer below. – usεr11852 Sep 12 at 23:47

2. Your book correctly emphasises the high learning rate $$\alpha$$ to show that the first learner matters a lot as the fit it provides is not strongly regularised. If $$\alpha$$ was low it would be plausible that even missing one of first learners it would not affect the overall fit that much. The overall effect of the first learners being the most impactful and the later learners making significantly diminishing contributions is well-known. A standard way to ameliorate that effect is/was actually using low $$\alpha$$ values; another prominent technique is DART trees where we "drop trees" at random in order of prevent over-specialisation of the base-learners (see Rashmi & Gilad-Bachrach (2015) DART: Dropouts meet Multiple Additive Regression Trees for more details).