Steps in gradient boosting algorithm

Question

Can some one please explain the 3rd step 2(c) in the below gradient boosting algorithm. I was under the impression, that the 2(c) computation is nothing but the mean of the corresponding terminal node (average of all the target values in the node- average of $r_{im}$, since $r_{im}$ is the target).

What parameter of GBM does gradient descent update after calculating gradient of loss function?

Also, isn't $f_{m-1}(x_i)$ assigned to $\gamma$ (a constant, in step 1) ? Not sure, why we are adding $f_{m-1}(x_i)$ to $\gamma$ which is like $2*\gamma$ in 2(c). Why are we using $f_{m-1}(x_i)$ and $\gamma$ and $L$, instead of mean of $r_{im}$ of the node, in step 2(c)

Answer 1

Let's start them from the beginning of the algorithm:

1. In step 1 we cannot assign $f_{m-1}(x_i)$ to anything as we have yet to estimate $f$. We just set it as the mean of the $y_i$ across all the samples (as we have yet to define any regions $R_j$).
2. • Your intuition about step 2(c) is correct. Just note that because of the existence of $f_{m-1}(x_i)$, $\gamma_{jm}$ will be expressed in terms of the residuals $r_{im}$. We effectively do use the "mean of $r_{im}$ of the node" as you say. Just that mean of that residuals is within the region $j$.
   • We are not adding $2\gamma$ because $f_{m−1}(x_i)$ is extremely unlikely to equal $\gamma$; $\gamma$ is effectively in the scale of $r$ as it is "a mean of residuals", $f_{m−1}(x_i)$ on the other hand is on the scale of the response variable $y$.

3. Note that in step 2(d) we are adding $\gamma_{jm}$ in our estimates of $x_i$ only if $x_i$ is within $R_{jm}$. Notice that the summation is across the $J$ regions.