Main concept of AdaBoost is that on each iteration algorithm learns what samples were difficult to classify and increases weights of these samples, while decreasing weights of those that were easy to classify. That's where the name adaptive boosting comes from.

My question: What makes AdaBoost algorithm so different comparing with general Gradient Boosting. Is exponential loss the only reason that leads to that "adaptive" algorithm concept?

  • 1
    $\begingroup$ Good question (+1), not at all "stupid", shows appreciation of the methodology. $\endgroup$ – usεr11852 Oct 5 '19 at 22:36

The main difference between AdaBoost and other "generic" boosting algorithms is that AdaBoost uses the (deviance) residuals as weights while "generic" gradient boosting algorithms use the residuals as the learning target itself.

This different between AdaBoost and other "generic" Gradient Boosting Machine (GBM) methodologies is more prominent when we examine a "generic" GBM as an additive model where we find the solution iteratively via the Backfitting algorithm (one can see Elements of Statistical Learning, Hastie et al. (2009) Ch. 10.2 "Boosting Fits an Additive Model" for the relation between boosting and additive models in more detail). To that extent you can look into LogitBoost as it effectively is the one that bridges AdaBoost and the "generic" GBM framework. The "LogitBoost paper" (Additive logistic regression: a statistical view of boosting, Friedman et al. (2000)) has a specific section (Sect. 4, AdaBoost: an additive logistic regression model) that focuses on the interpretation of AdaBoost as a stage-wise estimation procedure for fitting an additive logistic regression model. It focuses on how AdaBoost minimises the expectation of the exponential loss $E\{ e^{-y F(x)} \}$ via an iterative approach.

  • $\begingroup$ Thank you for your answer! I'm still a bit confused. First of all, I assume that residuals and antigradient are the same things, then in GBM on each iteration we find base learner as $b_t := \arg\min_b \sum_{i=1}^{l}(b(x_i)+L'(f_i, \space y_i))^2$. Thus, the part "use the residuals as the learning target itself" is clear for me, what is not clear is "uses the residuals as weights". In AdaBoost we update weights on iteration T as: $w_i := w_i exp(-y_i \alpha_T b_T(x_i)), i=1,...,l$, where previous $w_i = exp(-y_i\sum_{t=1}^{T-1}\alpha_tb_t(x_i))$. What must be treated as residuals here? $\endgroup$ – Egor Epishin Oct 6 '19 at 8:46
  • 1
    $\begingroup$ They residuals come in $\alpha_t = \frac{1}{2}\ln(\frac{1-\epsilon_t}{\epsilon_t})$ where they act as a learning rate scaler for the classifier. $\endgroup$ – usεr11852 Oct 6 '19 at 8:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.