3
$\begingroup$

I have written some code which uses AdaBoost to generate a set of weak classifiers. I'm finding, however, that when I use the resulting strong classifier to classify examples, it seems that almost every weak classifier classifies an example the same way. Either they all think it's negative, or they all think it's positive. As a result, instead of having a nice confidence measure at the end of the classification, I get an essentially binary value (either positive and > 1 or negative and < -1), and nothing really in between.

I'm using a weighted linear least squares weak classifier. When I examine the solved coefficients, I notice that they're very, very close in value among all of the weak classifiers. This seems strange to me, because I would expect that the AdaBoost method of weighting the difficult examples would cause the weighted linear least squares results to vary dramatically to 'fix' the difficult examples.

What is this telling me about my training set? Is it saying that the data are not linearly separable? Could there be a problem with my weighting scheme? Note also that there may be outliers in the positive and negative example set. Could this be a root cause?

I'm hoping somebody could provide some insight into how it could be that the weighted least squares solutions all seem to be almost identical.

Thanks!

Improve this question
$\endgroup$
3
$\begingroup$

It could simply be that the optimal solution to the classification problem is a linear discriminant and there is sufficient data to constrain the value of the weights very tightly. In that situation AdaBoost won't help you as a linear classifier is already very close to the Bayes optimal solution.

Improve this answer
$\endgroup$
  • $\begingroup$ Hi Dikran. I should have mentioned that the performance of the weighted least squares solution was not overly good (generally 25% error at best). I don't believe the reason is that the data is easily linearly discriminable. $\endgroup$ – aardvarkk Nov 22 '11 at 19:52
  • $\begingroup$ What is the best error rate you can get with any other method? Is there evidence that the Bayes error is less than 25%? $\endgroup$ – Dikran Marsupial Nov 23 '11 at 17:05
2
$\begingroup$

(I have just a bare understanding of AdaBoost, please correct/downvote if I'm wrong!)

You didn't write anything about the performance of your classifier. Did you actually test it? If it does well, that tells me that your "weak" learners can easily learn the statistics and therefore already perform well on their own, thus the broad consensus among them. As you observed, AdaBoost doesn't gain you much in this case. What you could do is

  • Use just one weak learner as a strong learner
  • Use computationally cheaper and thus weaker learners and then use AdaBoost to get a faster strong learner from them.
Improve this answer
$\endgroup$
  • $\begingroup$ Please see my comment on Dikran's answer. $\endgroup$ – aardvarkk Nov 22 '11 at 19:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.