I know the basic overview of how gradient boosting trees work but i am finding it hard to figure out the use of gradient in gradient boosting. My questions may seem stupid but it would be great if anyone can help me out !. So my question is

P.S : It would be even more helpful if someone can explain me with less math

– jld
Apr 4, 2018 at 23:51
• @Chaconne Thanks for the link. I have edited the question now . I am more interested in " what is the use of gradient in gradient boosting and why is it used"? Apr 5, 2018 at 1:29
• you should definitely give this a read: blog.kaggle.com/2017/01/23/…
– jld
Apr 5, 2018 at 6:41

Suppose you have a true value $y$ and a predicted value $\hat{y}$. The predicted value is constructed from some existing trees. Then you are trying to construct the next tree which gives a prediction $z$. Then your final prediction will be $\hat{y}+z$. The correct choice of $z$ is $z = y - \hat{y}$. Therefore, you are now constructing trees to predict $y - \hat{y}$.
It turns out this is a special case of gradient boosting when your loss function is $L = \frac{1}{2} (y - \hat{y})^2$, and your prediction target for this new tree is the gradient of this loss function as $y - \hat{y} = - \frac{\partial L}{\partial \hat{y}}$.
In a more formal definition, if you already have a prediction $\hat{y}$, and you are trying to add a new prediction $z$ from a new tree to it, then the loss function can be expanded by Taylor's expansion near $\hat{y}$ as $$L = L_0 + \frac{\partial L}{\partial \hat{y}} z$$ With the spirit of gradient descent, we want $z$ to be along negative gradient direction, hence $z \sim - \frac{\partial L}{\partial \hat{y}}$.
• Please add explanations of $L$ and $L_0$ e.g "where L is the loss-function and L0 is the loss-function in a given point" Mar 27, 2022 at 20:41