In the Gradient Boosting Tree algorithm, as described in https://en.wikipedia.org/wiki/Gradient_boosting#Gradient_tree_boosting, we update the previous model $F_m$ by adding the results $h_m$ of the decision tree applied to the residuals multiplied by a $\gamma$ coefficient. Why don't we fit the decision tree on $F_m - \eta r_m$, with $r_m$ the residuals and $\eta$ a learning rate ? The results would give us a new function $F_{m+1}$.

  • $\begingroup$ You are correct that these results would give us a new function $F_{m+1}$. $\gamma$ is used to make control the relative step among all tree regions. It is actually post-multiplied by $\nu$ in most cases. Side-note: Wikipedia is great but because it explore many different implementations, it is a bit "jumpy" and hard to follow. I suggest, first to focus on single source expositions (e.g. the Hastie el al. ESL book, Chapt. 10, or Shapiro and Freund's early Adaboost papers) and complement the parts that are a bit fuzzy with Wikipedia rather than vice versa. $\endgroup$ – usεr11852 says Reinstate Monic Dec 8 '18 at 9:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.