1
$\begingroup$

Can some one please explain the 3rd step 2(c) in the below gradient boosting algorithm. I was under the impression, that the 2(c) computation is nothing but the mean of the corresponding terminal node (average of all the target values in the node- average of $r_{im}$, since $r_{im}$ is the target).

What parameter of GBM does gradient descent update after calculating gradient of loss function?

Also, isn't $f_{m-1}(x_i)$ assigned to $\gamma$ (a constant, in step 1) ? Not sure, why we are adding $f_{m-1}(x_i)$ to $\gamma$ which is like $2*\gamma$ in 2(c). Why are we using $f_{m-1}(x_i)$ and $\gamma$ and $L$, instead of mean of $r_{im}$ of the node, in step 2(c)

$\endgroup$
2
  • 1
    $\begingroup$ Can you please check that the latexing of the Maths in your post has not changed your question? Also, did you mean to type a question in the end of the first paragraph? $\endgroup$ – usεr11852 Mar 3 '20 at 22:43
  • $\begingroup$ Thank you for the latex, no not intending for any question in first paragraph $\endgroup$ – tjt Mar 4 '20 at 4:15
1
$\begingroup$

Let's start them from the beginning of the algorithm:

  1. In step 1 we cannot assign $f_{m-1}(x_i)$ to anything as we have yet to estimate $f$. We just set it as the mean of the $y_i$ across all the samples (as we have yet to define any regions $R_j$).
    • Your intuition about step 2(c) is correct. Just note that because of the existence of $f_{m-1}(x_i)$, $\gamma_{jm}$ will be expressed in terms of the residuals $r_{im}$. We effectively do use the "mean of $r_{im}$ of the node" as you say. Just that mean of that residuals is within the region $j$.
    • We are not adding $2\gamma$ because $f_{m−1}(x_i)$ is extremely unlikely to equal $\gamma$; $\gamma$ is effectively in the scale of $r$ as it is "a mean of residuals", $f_{m−1}(x_i)$ on the other hand is on the scale of the response variable $y$.
  2. Note that in step 2(d) we are adding $\gamma_{jm}$ in our estimates of $x_i$ only if $x_i$ is within $R_{jm}$. Notice that the summation is across the $J$ regions.

$\endgroup$
5
  • $\begingroup$ Thank you so much for your response and remembering this. I get step 1, fm−1(xi) is avg(yi). Can you please explain if possible, how is mean of rim of terminal region j= the right hand side of 2(c)? Because fm−1(xi) =gamma = mean of yi [for all i] (from step 1); in 2(c) isn't the RHS same as argmin sigma L(yi [i only for x in node], 2*gamma )? if fm−1(xi) =gamma , why are we adding it twice in 2(c) ? $\endgroup$ – tjt Mar 4 '20 at 4:08
  • $\begingroup$ Also, by "Just that mean is within the region j" in your answer, you mean mean of all the values in the terminal region ( or terminal node) j, right ? just trying to understand, if it is different from my statement "mean of rim of the node" . Thank you so much for taking the time to answer. $\endgroup$ – tjt Mar 4 '20 at 4:18
  • 1
    $\begingroup$ I am happy to help. Please see the additional point particular to your query about "$2\gamma$. I also now clarify that we are talking about a "mean of (that) residuals". I hope these answer your question. $\endgroup$ – usεr11852 Mar 5 '20 at 0:27
  • $\begingroup$ thank you. I thought in the step 1, (for first iteration m=1) we are initializing fm-1(x) = gamma [which minimizes the loss between yi and gamma], is that not the case ? If that's not case, can you please tell me where the gamma in the RHS of 2(c) is coming from, and what it is (I believe gamma in step is mean of yi. But not sure of gamma in 2(c)). I am interested in the first iteration and looking into this algorithm to explain the first iteration. Also you said "γjm will be expressed in terms of the residuals rim", but I dont see rim in RHS of 2(c) $\endgroup$ – tjt Mar 5 '20 at 1:09
  • 1
    $\begingroup$ 1. At step 1 we have $m=0$. 2. As mentioned within the iterations $\gamma$ is the mean of the residuals $r$ in each corresponding region. 3. The RHS of 2(c) has $L$ which is directly associated with $r$. $\endgroup$ – usεr11852 Mar 5 '20 at 1:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.