Suppose we've trained a GBDT model with 100 trees with a fairly high learning rate. Consider two cases:

  1. We drop the first tree in the model

  2. We drop the last tree in the model

We then compare models performance on the train set.

I wrote that in case 1 performance will drop more than in case 2 b/c a high learning rate means less corrections are made with each new tree so presumably the last tree did not contribute that many corrections to the prediction.

The answer in the book is: In the case1 performance will drop more than in the case2. In GBDT model we have sequence of trees, each improve predictions of all previous. So, if we drop first tree — sum of all the rest trees will be biased and overall performance should drop. If we drop the last tree -- sum of all previous tree won't be affected, so performance will change insignificantly (in case we have enough trees).

My question is, is my explanation wrong? Is the learning rate irrelevant in this case? Also, the question italicizes "train set," why is this distinction important?

  • $\begingroup$ This is a fun little question! (+1) Please see my answer below. $\endgroup$ – usεr11852 Sep 12 at 23:47

The explanation provided in your book is correct.

Let's see now your sub-questions:

  1. (Preliminary point) Gradient Boosting (GB) Machines are iterative ensemble learners. Within the GB framework each new base learner tries to compensate for the deficiencies of the existing base-learners in the ensemble. This contrasts how other ensemble methods work where the predictions of the base learners are averaged together (random forests being the obvious example).

  2. Your book correctly emphasises the high learning rate $\alpha$ to show that the first learner matters a lot as the fit it provides is not strongly regularised. If $\alpha$ was low it would be plausible that even missing one of first learners it would not affect the overall fit that much. The overall effect of the first learners being the most impactful and the later learners making significantly diminishing contributions is well-known. A standard way to ameliorate that effect is/was actually using low $\alpha$ values; another prominent technique is DART trees where we "drop trees" at random in order of prevent over-specialisation of the base-learners (see Rashmi & Gilad-Bachrach (2015) DART: Dropouts meet Multiple Additive Regression Trees for more details).

  3. Your book correctly emphasises the training set performance to show you that the generalisability of model performance is not the main issue here. Usually when we evaluate a model's performance we have either a separate test set or some validation schema that is performed to emulate how well our model would perform in unseen data. Training set performance is how good we fit our known data. Training set performance is not indicative how generalisable our model is but it is indicative of how well we fit our known response. By focusing on the training set, the writer tries to remove any worries that performance is due to not fitting unseen data adequately.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.