Regarding boosting in the context of machine learning. One definition I have encountered talks about turning multiple weak learners into one strong learner, and another talks about starting with a prediction and iteratively improving it by learning predictors for residuals (such as gradient boosting).
The questions I have are:
Which definition is more accurate? (or are they equivalent?)
Does the second simply implement the first?
What is the formal definition in the literature?
Both definitions are reasonable working definitions and equally valid as they capture the iterative nature of boosting algorithms. I view the first one as more general because the iterative nature of boosting was not strictly speaking a requirement early on. Originally boosting was defined with respect to PAC learning in Schapire (1990) "The strength of weak learnability". To quote directly from the 2012 monograph "Boosting: Foundations and Algorithms" by Schapire and Freund: "Boosting has its roots in a theoretical framework for studying machine learning called the PAC model, proposed by Valiant, (...)" and "the key idea behind boosting is to choose training sets for the base learner in such a fashion as to force it to infer something new about the data each time it is called."
Not necessarily. For example, AdaBoost learns predictors for the sample points weighted by the residuals, not for the residuals (as would be the case for gradient boosting for example). That said, both gradient boosting and AdaBoost utilise weak learners to iteratively update their overall predictions "turning multiple weak learners into one strong learner". CV.SE has a very good thread on: In boosting, why are the learners "weak"? that discusses this matter in more detail.
There is no good concise formal definition. Since Breiman's 1998 "Arcing classifier (with discussion and a rejoinder by the author)" we have moved away from the ensemble methods view to the functional gradient descent view; Friedman's 2001 "Greedy function approximation: A gradient boosting machine" being pretty much the theoretical blue-print for all boosting algorithms after it. This numerical optimization in function space necessitates looking at boosting as an iterative procedure; the "incremental functions" learned are the "boosts" and we thus have a "stage-wise" update where each stage uses a suitable approximation to our loss function $L$ to search for its next "boost".
Something like: "Boosting refers to the greedy stage-wise updates done to a prediction function $F$ using weak learners $h$ in order to minimise a specified loss function $L$." is probably the most informal and succinct one can think of... (i.e. trying to put Eqs (9 & 10) from Friedman's 2001 into words.)
