I tried using AdaBoost for my classification which is for emotion classification. Without boosting, Random Forest algorithm gave me 42.41% of accuracy. But when I applied AdaBoost along with Random Forest as the base classifier, it reduced the accuracy to 38.61%.

As I learned and according to here, if it cannot improve the accuracy, the level without boosting should be given as the output.

So how to explain my situation?


Let me edit the question for you.

I tried several algorithms with my dataset with WEKA. I tried my dataset with and without boosting. In the following graph, my accuracies are shown.

enter image description here

As you can see, SMO , Naive bayes and LibSVM has behaved according to the theories.(If boosting cannot improve the accuracy, it will remain same). But some misbehavior can be seen with the Random Forest. That is where I am confused.

  • $\begingroup$ I don't think using adaBoost to boost randomForest(RF) would be a conventional approach, that would become a boosted ensemble of ensembles of bagged decision trees. Normally adaBoost is used with at weak learner, such as a single decision tree or maybe logistic regression. Try pure gradient boosting instead, (gbm or xgboost in R). $\endgroup$ Dec 9 '15 at 11:24
  • $\begingroup$ Why are you boosting a Random Forest in the first place? There's a lot we still don't really understand about AdaBoost, and the literature (at least what I've read) is conflicted on how it's supposed to behave in general. The answer you linked specifically talks about "classical AdaBoost" and "weak learners." Using a Random Forest as the base learner does not fit these criteria. $\endgroup$ Dec 12 '15 at 2:27
  • $\begingroup$ Then the short answer for my question would be, since random forest is already a strong classifier, it does not really necessary to behave the way what I have linked. Is that? $\endgroup$
    – vigamage
    Dec 12 '15 at 2:35

First of all, make sure that the objective function of the boosting is decreasing over time (otherwise, most likely, there is a bug in the code!).

Two possibilities comes to mind:

  1. overfitting to training data
  2. AdaBoost minimizes an exponential loss. If you are measuring the performance using some other measure (e.g. 0-1 classification loss) then the mis-match between training and the test loss may be the reason. This becomes more likely if you have outliers in your training data (that is, samples that are either labeled incorrectly in the training data or are hard for your classifier)
  • $\begingroup$ this is something I tried using WEKA. not using a code written by me. I tried several algorithms with my dataset. For SMO and Naive Bayes, the accuracies they showed without boosting remained even after the boosting is applied. That is according to the fact that I have mentioned in the question. But for Random forest only, the accuracy got decreased. That is my problem, $\endgroup$
    – vigamage
    Dec 12 '15 at 2:13
  • 2
    $\begingroup$ @vigamage accuracy is not the same thing as exponential loss. $\endgroup$ Dec 12 '15 at 2:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.