As I understand the idea of gradient boosting in the (m+1)-th step we take the partial derivatives of the loss with respect to our new parameters $f^{[m]}(x^{(i)})$: $\tilde{y}^{(i)}=-\frac{\partial (\sum_{i=1}^nL(y^{(i)}, f^{[m]}(x^{(i)})))}{\partial f^{[m]}(x^{(i)})}$

and afterwards approximate this gradient step with a function $f_\theta$ from our hypothesis space that comes close to this gradient step, i.e. $\tilde{f}= \text{argmin}_{f \in H}\sum_{i=1}^n\left[\tilde{y}^{(i)}- f_{\theta}(x^{(i)}) \right] ^2$

we then multiply $\tilde{f}$ by some constant (depending on the learning rate) to get $f^{[m+1]}$

Now my question is about the scenario in which our loss is not $0.5(y-f(x))^2$ but some other arbitrary loss function $L^*$. Then $\tilde{y}^{(i)} \neq (y^{(i)}-f^{[m]}(x^{(i)}))$. According to algorithm 1 in the Friedman paper (page 1193) we would then still minimize $\sum_{i=1}^n\left[\tilde{y}^{(i)}- f_{\theta}(x^{(i)}) \right] ^2$ in order to get our new function.

My question is why we would not simply minimize $\sum_{i=1}^nL(y^{(i)}-f^{[m]}(x^{(i)}), f_\theta(x^{(i)}))$

Is this only because the approach as in algorithm 1 by Friedman is cheaper or is there another reason?

  • $\begingroup$ xgboost.readthedocs.io/en/latest/tutorials/model.html In the case of a classification problem, xgboost uses cross-entropy loss as a default. The derivation of how to update weights is a straightforward application of Taylor approximations; you can use the same strategy to optimize arbitrary loss functions. $\endgroup$ – Reinstate Monica Oct 1 at 11:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.