# Is exponential loss function the only reason for AdaBoost being adaptive algorithm?

Main concept of AdaBoost is that on each iteration algorithm learns what samples were difficult to classify and increases weights of these samples, while decreasing weights of those that were easy to classify. That's where the name adaptive boosting comes from.

My question: What makes AdaBoost algorithm so different comparing with general Gradient Boosting. Is exponential loss the only reason that leads to that "adaptive" algorithm concept?

• Good question (+1), not at all "stupid", shows appreciation of the methodology. Oct 5, 2019 at 22:36

This different between AdaBoost and other "generic" Gradient Boosting Machine (GBM) methodologies is more prominent when we examine a "generic" GBM as an additive model where we find the solution iteratively via the Backfitting algorithm (one can see Elements of Statistical Learning, Hastie et al. (2009) Ch. 10.2 "Boosting Fits an Additive Model" for the relation between boosting and additive models in more detail). To that extent you can look into LogitBoost as it effectively is the one that bridges AdaBoost and the "generic" GBM framework. The "LogitBoost paper" (Additive logistic regression: a statistical view of boosting, Friedman et al. (2000)) has a specific section (Sect. 4, AdaBoost: an additive logistic regression model) that focuses on the interpretation of AdaBoost as a stage-wise estimation procedure for fitting an additive logistic regression model. It focuses on how AdaBoost minimises the expectation of the exponential loss $$E\{ e^{-y F(x)} \}$$ via an iterative approach.
• Thank you for your answer! I'm still a bit confused. First of all, I assume that residuals and antigradient are the same things, then in GBM on each iteration we find base learner as $b_t := \arg\min_b \sum_{i=1}^{l}(b(x_i)+L'(f_i, \space y_i))^2$. Thus, the part "use the residuals as the learning target itself" is clear for me, what is not clear is "uses the residuals as weights". In AdaBoost we update weights on iteration T as: $w_i := w_i exp(-y_i \alpha_T b_T(x_i)), i=1,...,l$, where previous $w_i = exp(-y_i\sum_{t=1}^{T-1}\alpha_tb_t(x_i))$. What must be treated as residuals here? Oct 6, 2019 at 8:46
• They residuals come in $\alpha_t = \frac{1}{2}\ln(\frac{1-\epsilon_t}{\epsilon_t})$ where they act as a learning rate scaler for the classifier. Oct 6, 2019 at 8:50