Reading through this overview of boosted trees, I'm having trouble understanding how the second line was derived.

$$ Obj(t)=\sum_1^n{loss(y_{i} - \hat{y}_i^{(t)})} + \sum_1^t{\Omega(f_i)} \\ = \sum_1^n{loss(y_{i} - (\hat{y}_i^{(t-1)} + f_t(x_i)))} + \Omega(f_t) + Constant $$

Why do we ignore the regularization of the other functions? And is the "Constant" term the irreducible error?

Also, in the equation following the one above, we use MSE as the loss function which results in:

$$ Obj(t)=\sum_1^n{(y_{i} - (\hat{y}_i^{(t-1)} + f_t(x_i))^2} + \sum_1^t{\Omega(f_i)} \\ = \sum_1^n{[2(\hat{y}_i^{(t-1)} - y_i)f_t(x_i) + f_t(x_i)^2]} + \Omega(f_t) + Constant $$

Why does the $ ( \hat{y}_i^{(t-1)} - y_i )^2 $ term disappear in the expansion?


I think you already know that the first substitution is the definition

$$\hat y_i^{(t)} = y_{i}^{(t-1)} + f_t(x_i)$$

Observe that, at this point in the discussion, we are attempting to fit the weak learner $f_t$, and all the previous weak learners $f_{t-1}, f_{t-2}, \ldots$ have already been fit, and are hence fixed. This means that any function of them, in particular $\Omega(f_{i-1}), \Omega(f_{i-2}), \ldots$ are also fixed. So breaking up the sum

$$ \sum_1^t \Omega(f_i) = \underbrace{\Omega(f_t)}_{\text{Currently up for optimization}} + \underbrace{\sum_1^{t-1} \Omega(f_t)}_{\text{Already fixed}}$$

The docs are simply making this observation, and replacing the second sum with "constant".

  • $\begingroup$ This makes sense. Thank you. I edited the above question to include a clarification for the next part, which I initially thought would follow from understanding what happened to the regularizations. $\endgroup$ – Duck Jan 22 '16 at 16:09
  • $\begingroup$ I'll add it to my answer in detail, but quickly, that term is absorbed into "constant" for the same reason, as $\hat y^{(t-1)}_i$ is fixed when you are fitting the $t$'th weak learner. $\endgroup$ – Matthew Drury Jan 22 '16 at 16:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.