With gradient boosting for regression, there are 2 loss functions, i.e:

  1. a custom loss function that we calculate the gradient for: $L(y_i,\hat{y_i})$
  2. the loss function used by the tree that fits the gradient $\triangledown_{\hat{y}} L(y,\hat{y})$, which is always squared loss


A possible resolution to this dilemma is to induce a tree T(x;$\Theta_m$) at the mth iteration whose predictions $t_m$ are as close as possible to the negative gradient. Using squared error to measure closeness, this leads us to $$ \tilde{\Theta}_m = \underset{\Theta}{\arg \min} \sum_{i=1}^{N}(-g_{im} - T(x_i;\Theta))^2 \tag{10.37}$$. That is, one fits the tree T to the negative gradient values (10.35) by least squares. (Hastie, Tibshirani, Friedman. The Elements of Statistical Learning, 2nd Ed)

Why not use the custom function for both? Would there be any disadvantages? The question was already asked in Non L2 loss-function in gradient boosting, but I'm not sure about the answer that states that this loss function has least variance given unbiasedness. This is true for OLS if the model is correctly specified, but for trees too? Sketches of proofs or keywords are much appreciated.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.