Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 2 down vote favorite

I run the cross-validation experiment for a given data set, and tried two different approaches: one is based on SVM, another is based on SVM plus Adaboost. But the confusion matrix for two experiments are exactly the same. I am confused on how to explain this kind of result. Adaboost is supposed to start with a weak classifier, but how to determine whether a classifier is weak?

share|improve this question
How do you apply AdaBoost to SVM, do you change the weights of points in optimized function every time? – sjm.majewski Aug 1 '12 at 11:53

1 Answer

up vote 1 down vote accepted

I can answer the second question. A weak classifier for AdaBoost means a classifier with low Variance (and consequently high bias) in Bias-Variance decomposition. The reason is, that combining classifiers increases variance, so when you start with high variance classifiers, you're very likely to overfit.

share|improve this answer
I think your definition of a weak classifier is accurate with regard to the earliest papers on boosting, but that low variance isn't necessarily a good thing for base classifiers. For example, Schapire et al. said in 1997 that "The original goal of boosting was to reduce the error of so-called “weak” learning algorithms which tend to have very large bias," but that boosting high-variance classifiers also worked, which is why boosting trees is so popular these days. cc.gatech.edu/~isbell/tutorials/boostingmargins.pdf – David J. Harris Aug 1 '12 at 17:13
Also, combining classifiers--especially if they're unstable--will usually decrease the variance, not increase it as you stated. That's why bagging works, as discussed in the same pdf I just linked. – David J. Harris Aug 1 '12 at 17:16
I haven't yet read the whole paper, but it's excellent and conveys some really usefull information, thank you. I'll update my answer regarding to it when I finish. – sjm.majewski Aug 5 '12 at 10:13

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.