In my book on statistical learning, an algorithm for boosting for regression trees is described. They have the main step of the algorithm as:

$\hat{f}(x) \leftarrow f(x) + \lambda \hat{f}^b(x)$,

where $\lambda$ is the shrinkage parameter, and $\hat{f}^{b}$ represents the recently computed boosted tree.

My question is how to derive the following rule for the update of the residuals:

$r_i \leftarrow r_i - \lambda \hat{f}^{b}(x)$

I've been stuck on this for quite a while. It seems simple, but I cannot get it.

  • $\begingroup$ I have been under the impression, perhaps falsely, that the residual for a given iteration is just $y - \hat{f}(x)$. In other words, the residual for the current ensemble of boosted trees. Are you sure that it's more complicated than that? $\endgroup$ – generic_user Aug 20 '18 at 17:43
  • $\begingroup$ Hi. Yes, according to my textbook which is "Introduction to Statistical Learning" by James and Witten. It is Algorithm 8.2 in the textbook on page 322. $\endgroup$ – Thomas Moore Aug 20 '18 at 18:06
  • 1
    $\begingroup$ OK, well maybe that explains why my hand-coded attempt at replicating it didn't work so well! $\endgroup$ – generic_user Aug 20 '18 at 18:10
  • 1
    $\begingroup$ If $r_i = y_i - \hat{y}_i$ then $\text{new} r_i = y_i - \hat{f} = y_i - (f + \lambda f^b) = \text{old} r_i - \lambda f^b$ no? $\endgroup$ – Fabian Werner Aug 20 '18 at 18:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.