3
$\begingroup$

I am working on a binary classification case and comparing the performance of different classifiers.Testing the performance of adaboost algorithm (with decision tree as its base classifier) against SVM on multiple data sets I found that the boosting algorithm performs better.

The question I have is why this is happening? Is this because boosting always outperforms SVM? or it has something to do with the characteristics of my data set? I wonder if anyone can help me with some potential explanations of such finding.

$\endgroup$
6
  • $\begingroup$ generally better, but not always. but that is just based on past experience. sure there are many papers comparing. and depends greatly on how rigorously models developed (how was boosted model tuned, which kernels tried etc.) $\endgroup$
    – charles
    Aug 13 '14 at 1:13
  • $\begingroup$ Thanks. Would be great if you could share the link of couple of those articles. $\endgroup$
    – AliCivil
    Aug 13 '14 at 1:50
  • $\begingroup$ I was thinking of this paper: niculescu-mizil.org/papers/comparison.tr.pdf (+/-problematic) But google will produce more: lowrank.net/nikos/pubs/empirical.pdf I'm not convinced by these, hard thing to study, but have found that most practitioners will push for RF or boosting algorithms. $\endgroup$
    – charles
    Aug 13 '14 at 2:03
  • $\begingroup$ No problem. Not really a great answer, but hopefully useful. I also came across this while looking for stuff on heterogeneous ensembles: business-school.ed.ac.uk/waf/crc_archive/2013/42.pdf $\endgroup$
    – charles
    Aug 16 '14 at 2:29
  • $\begingroup$ You have not really been given proper answers though. One question certainly is if you use a non-linear kernel with the SVM and cross-validate the regularization parameter as well as the parameters of the kernel. $\endgroup$
    – pAt84
    Mar 5 '16 at 10:56
5
$\begingroup$

For me your question is part of a larger one involving classification algorithms in general. This paper Do we Need Hundreds of Classifiers to Solve Real World Classfication Problems?, gets at this more fundamental question in finding that random forests outperforms all other competitors. A link to an ungated copy is here:

http://jmlr.csail.mit.edu/papers/volume15/delgado14a/delgado14a.pdf

There are other flavors of boosting that haven't been mentioned such as GBM, the generalized boosted model, which has an R package:

https://github.com/gbm-developers/gbm

Also worth a mention is XGBoost or extreme gradient boosting, which some have found to out perform everything (and despite the Delgado paper's findings).

https://cran.r-project.org/web/packages/xgboost/xgboost.pdf

$\endgroup$
1
  • $\begingroup$ +1. And to emphasize, the OP's "always outperform" is problematic. Not just because of the No Free Lunch theorem, but because of specific issues like AdaBoost (and similar boosting algorithms) can perform terribly on data where the targets are mislabeled, while my guess is that SVMs would be more robust to that problem. $\endgroup$
    – Wayne
    Mar 5 '16 at 16:14
2
$\begingroup$

In short: Regular AdaBoost (decision stumps, not decision trees as weak learners) usually outperform linear SVMs. Of course this is dependent on your data, data normalization, etc, so this is certainly not a general rule. The reasoning should be fairly clear. AdaBoost can learn non-linear decision boundaries, which is almost always helpful, especially if your data can not be linearly separated. If it can, it is all just a question of empircal results and not much can be said about AdaBoost vs. SVM except that the SVM builds upon Statistical Learning Theory (SLT, http://videolectures.net/mlss09uk_shawe_taylor_lt/?q=machine%20learning%20summer%20school). I am not entirely sure that can be said for AdaBoost. Given that the SVM maximizes the margin between the point clouds it should, given linearly separable data, outperform AdaBoost since we would expect better generalization from the SVM than from AdaBoost at this point.

When the SVM uses a proper non-linear kernel that fits the data better and generalizes well, it usually outperforms AdaBoost.

AdaBoost is commonly used because it is very fast. Viola & Jones have used it 15 years ago to realize fast object detection, especially face recognition: https://www.cs.cmu.edu/~efros/courses/LBMV07/Papers/viola-cvpr-01.pdf

In addition, please see my comments on your question.

$\endgroup$
0
$\begingroup$

It looks like you may have used a linear SVM (i.e. an SVM with a linear kernel). Depending on the base learner, ADABoost can learn a non-linear boundary, so may perform better than the linear SVM if the data is not linearly separable. This of course depends on the characteristics of the dataset. See this page on wikipedia for more information about kernel methods.

$\endgroup$
1
  • $\begingroup$ I think the answer to the OP's question is implied here, but you aren't quite stating it explicitly. Can you spell out the answer to the OP's question? $\endgroup$ Oct 21 '15 at 17:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.