# Non L2 loss-function in gradient boosting

As I understand the idea of gradient boosting in the (m+1)-th step we take the partial derivatives of the loss with respect to our new parameters $$f^{[m]}(x^{(i)})$$: $$\tilde{y}^{(i)}=-\frac{\partial (\sum_{i=1}^nL(y^{(i)}, f^{[m]}(x^{(i)})))}{\partial f^{[m]}(x^{(i)})}$$

and afterwards approximate this gradient step with a function $$f_\theta$$ from our hypothesis space that comes close to this gradient step, i.e. $$\tilde{f}= \text{argmin}_{f \in H}\sum_{i=1}^n\left[\tilde{y}^{(i)}- f_{\theta}(x^{(i)}) \right] ^2$$

we then multiply $$\tilde{f}$$ by some constant (depending on the learning rate) to get $$f^{[m+1]}$$

Now my question is about the scenario in which our loss is not $$0.5(y-f(x))^2$$ but some other arbitrary loss function $$L^*$$. Then $$\tilde{y}^{(i)} \neq (y^{(i)}-f^{[m]}(x^{(i)}))$$. According to algorithm 1 in the Friedman paper (page 1193) we would then still minimize $$\sum_{i=1}^n\left[\tilde{y}^{(i)}- f_{\theta}(x^{(i)}) \right] ^2$$ in order to get our new function.

My question is why we would not simply minimize $$\sum_{i=1}^nL(y^{(i)}-f^{[m]}(x^{(i)}), f_\theta(x^{(i)}))$$

Is this only because the approach as in algorithm 1 by Friedman is cheaper or is there another reason?

• xgboost.readthedocs.io/en/latest/tutorials/model.html In the case of a classification problem, xgboost uses cross-entropy loss as a default. The derivation of how to update weights is a straightforward application of Taylor approximations; you can use the same strategy to optimize arbitrary loss functions. – Reinstate Monica Oct 1 at 11:31