14
$\begingroup$

As I've heard of the AdaBoost classifier repeatedly mentioned at work, I wanted to get a better feel for how it works and when one might want to use it. I've gone ahead and read a number of papers and tutorials on it which I found on Google, but there are aspects of the classifier which I'm still having trouble understanding:

  1. Most tutorials I've seen speak of AdaBoost as finding the best weighted combination of many classifiers. This makes sense to me. What does not make sense are implementations (i.e. MALLET) where AdaBoost seems to only accept one weak learner. How does this make any sense? If there's only one classifier provided to AdaBoost, shouldn't it just return back that same classifier with a weight of 1? How does it produce new classifiers from the first classifier?

  2. When would one actually want to use AdaBoost? I've read that it's supposed to be one of the best out-of-the-box classifiers, but when I try boosting a MaxEnt classifier I was getting f-scores of 70%+ with, AdaBoost murders it and gives me f-scores of something like 15% with very high recall and very low precision instead. So now I'm confused. When would I ever want to use AdaBoost? I'm looking for more of an intuitive rather than strictly statistical answer, if possible.

$\endgroup$
11
$\begingroup$

Adaboost can use multiple instances of the same classifier with different parameters. Thus, a previously linear classifier can be combined into nonlinear classifiers. Or, as the AdaBoost people like to put it, multiple weak learners can make one strong learner. A nice picture can be found here, on the bottom.

Basically, it goes as with any other learning algorithm: on some datasets it works, on some it doesn't. There sure are datasets out there, where it excels. And maybe you haven't chosen the right weak learner yet. Did you try logistic regression? Did you visualize how the decision boundaries evolve during adding of learners? Maybe you can tell what is going wrong.

$\endgroup$
  • $\begingroup$ (+1). Additional remark: Even the parameters of the weak learners to not differ, the behavior (i.e. what they predict) changes when the structure of the dataset (as done in boosting) changes. $\endgroup$ – steffen Mar 30 '11 at 9:09
  • $\begingroup$ MaxEnt is logistic regression. Visualizing how decision boundaries evolve would actually be quite useful. How can one go about doing that? Also, I still don't understand how AdaBoost still works when it's only given one feature set and one classifier type (as in MALLET). To me, this seems completely counter-intuitive. $\endgroup$ – YuliaPro Mar 30 '11 at 13:25
  • $\begingroup$ Just plot your dataset. Additionaly, plot a grid where you pick a color for each point how it's being classified by your system. This works only good for 2D, I guess. You could still plot the most important axes or something. If you only have one feature set and one classifier, it reduces to only one classifier. I was under the impression, that maximum entropy was a paradigm and not a concrete classifier. Anyway. $\endgroup$ – bayerj Mar 30 '11 at 15:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.