After applying AdaBoost to the Sonar data, I obtain something like this.

I dont understand that phenomenon, can anyone enlighten me?

  • 1
    It appears to be related to how confident AdaBoost is in correctly classifying the observations. See Schapiro and Freund's book, p16 has some introductory remarks that point to later chapters. – Christoph Hanck Sep 4 '15 at 15:13
up vote 4 down vote accepted

Exactly this question is answered by Schapire and Freund, et. al. in their publication

Boosting the Margin: A New Explanation for the Effectiveness of Voting Methods Robert E. Schapire, Yoav Freund, Peter Bartlett and Wee Sun Lee The Annals of Statistics Vol. 26, No. 5 (Oct., 1998), pp. 1651-1686

The takeaway from all their math is, that even when the training error is zero, the margin (= sample distance to decision boundary) is still improved by further boosting iterations. Also, among other things they show that there are lower upper bounds on the generalization error for a classifier with larger margin, which at least supports what you are seeing.

Your Answer


By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.