Gradient boosting and functional gradient descent Why in gradient boosting methods do we fit models to gradients rather than errors? I'm not too familiar with calculus of variations and functionals so could someone give an intuitive explanation for why models are applied to gradients rather than residuals? Pictures would be greatly appreciated!
 A: I'll briefly outline Boosting and Gradient Descent, I think it's important to fully understand these before attempting to understand Gradient Boosting.   
Boosting: The goal of boosting is to make a strong hypothesis by combining several weak hypotheses. Here the strength/weakness of a hypothesis is measured by some error measure. So at each iteration of your boosting algorithm, you want to add a hypothesis to your current combination to minimize the error. 
Gradient Descent: The goal of Gradient Descent is to solve the following problem. Given a function $f(x)$ find $x$ such that $f(x)$ is as small as possible. Gradient Descent does this by taking small steps in the negative direction of the gradient. I'd advice you try to do this for a simple function like $f(x)=x^2+3$ or something. The answer is obviously $x=0$, but I think it's very valuable to some value of $x$ (maybe -2) and follow the algorithm (see [1] for more info on Gradient Descent). 
Gradient Boosting: It turns out several boosting algorithms (including AdaBoost, LogitBoost and Arc-GV) can be generalized as Gradient Descent in functional space [2]. I tend to think of this as boosting algorithms are just ways of doing Gradient Descent. We have some error function, and we want to find a classifier that minimizes this error. This is done by iteratively taking small steps in the direction of the negative gradient, which corresponds to adding a weak hypothesis that is trained to approximate the negative gradient. 
[1] https://en.wikipedia.org/wiki/Gradient_descent
[2] https://papers.nips.cc/paper/1766-boosting-algorithms-as-gradient-descent.pdf
