I have found mentions of two advantages in using gradients instead of actual residuals:

1) Using gradients will allow us to plug in any loss function (not just mse) without having to change our base learners to make them compatible with the loss function.

2) It can be computationally infeasible (in the case of MAE, we would have to calculate the median in every split).

However, I don't understand these points, namely:

1) If we just calculate the residual and have the base learner fit on those values, how exactly would that be any more difficult than calculating the gradients and then fitting on those values?

2) Following on the above question, why exactly do we need to calculate the median in the MAE example instead of just getting the residuals?

Perhaps I don't understand the exact mechanism behind optimizing an individual loss function. It would be really great if someone could demonstrate exactly how MAE is done and in which step would using residuals have been infeasible.

  • 1
    $\begingroup$ What is your question, actually? Your title doesn't match the last paragraph, or either of the interior two questions... $\endgroup$ – jbowman May 14 '18 at 1:48
  • $\begingroup$ @jbowman my question is about what is the advantage of using gradients over residuals in boosting. The middle questions are elaborating on why I was not satisfied by the reasons I found in my search, and the last part provides a possible way to answer the question that I think will be helpful. $\endgroup$ – eyio May 14 '18 at 2:00
  • $\begingroup$ This is a very interesting question, +1. Can you please provide some references for the "mentions" you provide? Especially the second mention seems pretty weak. $\endgroup$ – usεr11852 Feb 2 at 21:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.