After applying AdaBoost to the Sonar data, I obtain something like this.

I dont understand that phenomenon, can anyone enlighten me?

  • 1
    $\begingroup$ It appears to be related to how confident AdaBoost is in correctly classifying the observations. See Schapiro and Freund's book, p16 has some introductory remarks that point to later chapters. $\endgroup$ – Christoph Hanck Sep 4 '15 at 15:13

Exactly this question is answered by Schapire and Freund, et. al. in their publication

Boosting the Margin: A New Explanation for the Effectiveness of Voting Methods Robert E. Schapire, Yoav Freund, Peter Bartlett and Wee Sun Lee The Annals of Statistics Vol. 26, No. 5 (Oct., 1998), pp. 1651-1686

The takeaway from all their math is, that even when the training error is zero, the margin (= sample distance to decision boundary) is still improved by further boosting iterations. Also, among other things they show that there are lower upper bounds on the generalization error for a classifier with larger margin, which at least supports what you are seeing.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.