I run the cross-validation experiment for a given data set, and tried two different approaches: one is based on SVM, another is based on SVM plus Adaboost. But the confusion matrix for two experiments are exactly the same. I am confused on how to explain this kind of result. Adaboost is supposed to start with a weak classifier, but how to determine whether a classifier is weak?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ How do you apply AdaBoost to SVM, do you change the weights of points in optimized function every time? $\endgroup$– sjm.majewskiCommented Aug 1, 2012 at 11:53
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
3
I can answer the second question. A weak classifier for AdaBoost means a classifier with low Variance (and consequently high bias) in Bias-Variance decomposition. The reason is, that combining classifiers increases variance, so when you start with high variance classifiers, you're very likely to overfit.
answered Jul 31, 2012 at 17:37
-
$\begingroup$ I think your definition of a weak classifier is accurate with regard to the earliest papers on boosting, but that low variance isn't necessarily a good thing for base classifiers. For example, Schapire et al. said in 1997 that "The original goal of boosting was to reduce the error of so-called “weak” learning algorithms which tend to have very large bias," but that boosting high-variance classifiers also worked, which is why boosting trees is so popular these days. cc.gatech.edu/~isbell/tutorials/boostingmargins.pdf $\endgroup$ Commented Aug 1, 2012 at 17:13
-
$\begingroup$ Also, combining classifiers--especially if they're unstable--will usually decrease the variance, not increase it as you stated. That's why bagging works, as discussed in the same pdf I just linked. $\endgroup$ Commented Aug 1, 2012 at 17:16
-
$\begingroup$ I haven't yet read the whole paper, but it's excellent and conveys some really usefull information, thank you. I'll update my answer regarding to it when I finish. $\endgroup$ Commented Aug 5, 2012 at 10:13