0
$\begingroup$

In optimization with gradient descent I update cost function parameters with respect to the negative gradients. In gradient boosted treed, I update the prediction function by calculating negative gradients and then fitting a tree to them. Why bother and go through the last step of fitting?

Just to add an example for illustration purposes: In the picture below I started with a simple linear regression (blue). Then I computed the gradients (with squard loss this equals the residuals) and fitted a tree to them (purple step-function below) and added it to the regression line to obtain my new prediction function. I could have just added the residuals instead, couldnt I?

Simple example

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

The point of fitting the negative gradient is to learn a mapping from input space to output space. That is, we want a function that can take any input and produce the corresponding output (in a way that approximates the training set well).

In your example, suppose you did simply add the residuals to the predicted output from the linear model. This would perfectly reproduce the outputs on the training set, and would almost certainly be overfitting. But, furthermore, you'd have no way to predict outputs for inputs that aren't part of the training set. By iteratively fitting the residuals, you obtain functions that can be evaluated for any input, and combined to produce the final output.

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thanks, great answer! But, i've also seen gradient descent for parameterized functions (like regression or log regression). For those animals I could produce updated prediction functions by evaluating at interim parameter estimates? $\endgroup$
    – laterstat
    Commented Nov 17, 2017 at 8:41
  • $\begingroup$ Not sure I understand the question in your comment $\endgroup$
    – user20160
    Commented Nov 17, 2017 at 8:47
  • $\begingroup$ Sorry! Given a specified cost function J(y,F(x,w)) for prediction function F(x,w) with features w and inputs x, I could iteratively update the parameters with negative gradients. And come up with a new prediction function every step of the way by just inserting the new parameters w. But I guess then I wouldnt have an ensemble. Irrespective of that, could I say that one of the differences between GBM and GD for those functions is that I update on derivatives with respect to parameters in one case and on derivatives with respect to estimation functions in the other $\endgroup$
    – laterstat
    Commented Nov 17, 2017 at 8:56
  • $\begingroup$ Yes, that sounds right (modulo some terminology). Alternatively, you could say they're similar in the sense that gradient boosting is performing gradient descent in function space. $\endgroup$
    – user20160
    Commented Nov 17, 2017 at 12:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.