I have found mentions of two advantages in using gradients instead of actual residuals:
1) Using gradients will allow us to plug in any loss function (not just mse) without having to change our base learners to make them compatible with the loss function.
2) It can be computationally infeasible (in the case of MAE, we would have to calculate the median in every split).
However, I don't understand these points, namely:
1) If we just calculate the residual and have the base learner fit on those values, how exactly would that be any more difficult than calculating the gradients and then fitting on those values?
2) Following on the above question, why exactly do we need to calculate the median in the MAE example instead of just getting the residuals?
Perhaps I don't understand the exact mechanism behind optimizing an individual loss function. It would be really great if someone could demonstrate exactly how MAE is done and in which step would using residuals have been infeasible.