I have written some code which uses AdaBoost to generate a set of weak classifiers. I'm finding, however, that when I use the resulting strong classifier to classify examples, it seems that almost every weak classifier classifies an example the same way. Either they all think it's negative, or they all think it's positive. As a result, instead of having a nice confidence measure at the end of the classification, I get an essentially binary value (either positive and > 1 or negative and < -1), and nothing really in between.
I'm using a weighted linear least squares weak classifier. When I examine the solved coefficients, I notice that they're very, very close in value among all of the weak classifiers. This seems strange to me, because I would expect that the AdaBoost method of weighting the difficult examples would cause the weighted linear least squares results to vary dramatically to 'fix' the difficult examples.
What is this telling me about my training set? Is it saying that the data are not linearly separable? Could there be a problem with my weighting scheme? Note also that there may be outliers in the positive and negative example set. Could this be a root cause?
I'm hoping somebody could provide some insight into how it could be that the weighted least squares solutions all seem to be almost identical.