2
$\begingroup$

The documentation on ensemble methods in Matlab explains different ensemble algorithms for classification and regression tasks. I have normalized the raw feature set and using the normalized data for training and testing.

My data is imbalanced and so I am interested to apply RUSBoost method. It is specifically used for such a scenario. There are some points which are not clear from the documentation and for which I could not find answers in other resources. Few points on ensemble learning cleared from this question on SO

However, I have confusions on applying boosting method with ensemble learning. Here are my questions:

  1. Can RUSBoost be applied without using Ensemble classifiers? Are boosting techniques- undersampling and oversampling methods applicable for ensemble classifiers only?
  2. Why only decision tree can be used with RUSBoost. I tried using other learners such as knn and svmtemplate but these throw errors.

Please correct me where I have misunderstood.

$\endgroup$

migrated from stackoverflow.com Jun 7 '18 at 17:11

This question came from our site for professional and enthusiast programmers.

1
$\begingroup$

Regarding question 1:

The RUSBoost implementation is part of the ensemble framework, so it is necessarily tied to ensembles. However, the idea behind RUSBoost can be applied without an ensemble.

Quoting from the description of RUSBoost

RUS stands for Random Under Sampling. The algorithm takes N, the number of members in the class with the fewest members in the training data, as the basic unit for sampling. Classes with more members are under sampled by taking only N observations of every class. In other words, if there are K classes, then, for each weak learner in the ensemble, RUSBoost takes a subset of the data with N observations from each of the K classes.

If we choose to ignore the boosting aspect of the implementation, this is simply applying undersampling to the bigger classes in your data, which is a known concept for tackling class imbalance and is independent of the classifier that you use. Note, however, that this procedure of course leads to less data being used by the classifier, which might impact the predicition quality.
Another approach to undersampling the other classes would be oversampling the minority class(es) , but as always each method has their drawbacks.

Regarding question 2:
From the documentation page about the ensemble learning framework

Except for Subspace, all boosting and bagging algorithms are based on decision tree learners. Subspace can use either discriminant analysis or k-nearest neighbor learners.

This is a restriction of Matlabs implementation, which makes sense because decision trees are very often used in ensemble methods. Some reasons are explained in this answer.

$\endgroup$
  • $\begingroup$ Thank you for your answer. I had earlier applied svm with Matlab's crossvalind function which gave poor results -- all predicted classes were incorrect. This motivated me to use RUSBoost with tree ensemble and it still performs poorly. Then I applied the SVM by carefully choosing the train and test without using the crossvalidation function in Matlab. SVM now gives better performance. Therefore, I am now confused as to what is the problem and the proper approach. $\endgroup$ – Srishti M Jun 8 '18 at 17:58
  • $\begingroup$ You might want to read up on crossvalidation and learning with imbalanced data before proceeding. You might want to have a look at the confusion matrix of your model and go from there. A question like "it's not working, please help" is somewhat hard to answer. $\endgroup$ – deemel Jun 8 '18 at 19:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.