For my classification, I use several algorithms available in WEKA, but with limited number of features. I got some accuracy levels with the algorithms I used and I tried improving the accuracies using Ensemble methods. I used Boosting and Bagging for that. But the outcome I got is strange.

Following figures show, the accuracies I got with and without bagging and boosting.

  1. Difference of accuracy level with and without Boosting. enter image description here

  2. Difference of accuracy with and without Bagging. enter image description here

What I expected to see after bagging and boosting was some improvements of the accuracy levels. But you can see that,

  1. when I used Random Forest algorithm with boosting, the accuracy has been decreased. I can understand that why accuracies do not get improved. But I do not know why they get decreased.
  2. In bagging, both Random forest and Naive Bayed have shown unexpected behaviors.

I did all these experiments in WEKA. I need to know why this happens? And also do they happen since I have a limited amount of features for my classification?

  • $\begingroup$ crude answer: You typically boost to lower the bias of the model (the error by lack of fit due to model structure cannot bend to fit the data structure). When you boost you risk to introduce overfitting which will lower accuracy estimated by cross-validation. If the model bias is already low, boosting is less likely to improve performance. Try to read chapter 10 in ' The Elements of Statistical Learning': web.stanford.edu/~hastie/local.ftp/Springer/OLD/… $\endgroup$ – Soren Havelund Welling Dec 14 '15 at 14:57
  • $\begingroup$ this question is very similar to your last question: stats.stackexchange.com/questions/185627/… $\endgroup$ – Soren Havelund Welling Dec 14 '15 at 15:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.