What to do when weak classifiers are almost identical in AdaBoost?

I have written some code which uses AdaBoost to generate a set of weak classifiers. I'm finding, however, that when I use the resulting strong classifier to classify examples, it seems that almost every weak classifier classifies an example the same way. Either they all think it's negative, or they all think it's positive. As a result, instead of having a nice confidence measure at the end of the classification, I get an essentially binary value (either positive and > 1 or negative and < -1), and nothing really in between.

I'm using a weighted linear least squares weak classifier. When I examine the solved coefficients, I notice that they're very, very close in value among all of the weak classifiers. This seems strange to me, because I would expect that the AdaBoost method of weighting the difficult examples would cause the weighted linear least squares results to vary dramatically to 'fix' the difficult examples.

What is this telling me about my training set? Is it saying that the data are not linearly separable? Could there be a problem with my weighting scheme? Note also that there may be outliers in the positive and negative example set. Could this be a root cause?

I'm hoping somebody could provide some insight into how it could be that the weighted least squares solutions all seem to be almost identical.

Thanks!

-

migrated from math.stackexchange.comNov 22 '11 at 14:53

This question came from our site for people studying math at any level and professionals in related fields.

It could simply be that the optimal solution to the classification problem is a linear discriminant and there is sufficient data to constrain the value of the weights very tightly. In that situation AdaBoost won't help you as a linear classifier is already very close to the Bayes optimal solution.

-
Hi Dikran. I should have mentioned that the performance of the weighted least squares solution was not overly good (generally 25% error at best). I don't believe the reason is that the data is easily linearly discriminable. –  aardvarkk Nov 22 '11 at 19:52
What is the best error rate you can get with any other method? Is there evidence that the Bayes error is less than 25%? –  Dikran Marsupial Nov 23 '11 at 17:05