2
$\begingroup$

With gradient boosting for regression, there are 2 loss functions, i.e:

  1. a custom loss function that we calculate the gradient for: $L(y_i,\hat{y_i})$
  2. the loss function used by the tree that fits the gradient $\triangledown_{\hat{y}} L(y,\hat{y})$, which is always squared loss

See:

A possible resolution to this dilemma is to induce a tree T(x;$\Theta_m$) at the mth iteration whose predictions $t_m$ are as close as possible to the negative gradient. Using squared error to measure closeness, this leads us to $$ \tilde{\Theta}_m = \underset{\Theta}{\arg \min} \sum_{i=1}^{N}(-g_{im} - T(x_i;\Theta))^2 \tag{10.37}$$. That is, one fits the tree T to the negative gradient values (10.35) by least squares. (Hastie, Tibshirani, Friedman. The Elements of Statistical Learning, 2nd Ed)

Why not use the custom function for both? Would there be any disadvantages? The question was already asked in Non L2 loss-function in gradient boosting, but I'm not sure about the answer that states that this loss function has least variance given unbiasedness. This is true for OLS if the model is correctly specified, but for trees too? Sketches of proofs or keywords are much appreciated.

Improve this question
$\endgroup$
1
  • $\begingroup$ A reason for using the squared error loss function is essentially that it is easy to optimize for most choices of $T$ - we do least squared on $g_{im}$, and assume a-priori that this is easy to do (which is usually the case). In the case of CART, you can even loop efficiently over the possible cutpoints, just tweaking the first and second empirical moments as observations move from one leaf node to another. This may not be the case with the custom loss function; it may be difficult to optimize over the space of trees efficiently if we cannot compute the optimal leaf node values efficiently. $\endgroup$
    – guy
    Sep 7, 2021 at 15:40

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.