Despite the resembling and other increasing data variability approaches, can the random forest "as an algorithm" be considered a good option for the unbalanced data classification?
$\begingroup$
$\endgroup$
6
-
1$\begingroup$ No. (Please be more specific in your question, as it stands it is too broad. You need to clarify your situation as well as what you mean by the statement "as an algorithm" - as opposed to something else?) $\endgroup$– usεr11852Oct 27, 2016 at 21:40
-
7$\begingroup$ @usεr11852 I don't think it's too broad at all -- it just has a one-word answer. $\endgroup$– shadowtalkerOct 27, 2016 at 23:34
-
$\begingroup$ What I meant by as an algorithm is compared to other classification tools such as SVM, logistic regression,....is RF considered a good option? $\endgroup$– mhdellaOct 28, 2016 at 8:20
-
$\begingroup$ Why not edit your question to show what comparators you are considering and what situation you envisage using your chosen method in? $\endgroup$– mdeweyOct 28, 2016 at 10:44
-
1$\begingroup$ @ssdecontrol: I am all for succinct answers; I rarely found one-word answers to be very enlightening though. Your own answer is a proof of that (as it is not one-worded :D ). $\endgroup$– usεr11852Oct 29, 2016 at 10:38
|
Show 1 more comment
2 Answers
$\begingroup$
$\endgroup$
5
Note: This post is fairly old, and might not be correct. Use it only as a starting point, not an authoritative answer.
The random forest model is built on decision trees, and decision trees are sensitive to class imbalance. Each tree is built on a "bag", and each bag is a uniform random sample from the data (with replacement). Therefore each tree will be biased in the same direction and magnitude (on average) by class imbalance.
However, several techniques exist for mitigating imbalance in classification tasks.
Some of these are general and apply to a wide variety of situations. Search for the unbalanced-classes tag on this SE site, and the class-imbalance tag on the Data Science SE site.
In addition, random forests are amenable to at least two kinds of class weighting. The first technique is to weight the tree splitting criterion (For information on how this works, see https://datascience.stackexchange.com/a/56260/1156). The other technique is to either oversample or undersample data points during the bootstrap sampling process.
In Python, weighted tree splitting is implemented in the Scikit-learn class RandomForestClassifier, as the class_weight parameter. Weighted bootstrap sampling is implemented in the Imbalanced-learn class BalancedRandomForestClassifier. Note that the Imbalanced-learn BalancedRandomForestClassifier also supports the same class_weight parameter as the Scikit-learn RandomForestClassifier.
In R, both techniques are implemented in the Ranger, in the main ranger function, as the class.weights, case.weights, and sample.fraction parameters. See https://stats.stackexchange.com/a/287849/36229 for a usage example; there is helpful information in the other answers on that same question as well.
Apparently, in every extreme cases of class imbalance, you might need to adjust the minimum node size or other "detailed" parameters to get the model to work at all. See, e.g. https://stackoverflow.com/a/8704882/2954547.
answered Oct 27, 2016 at 23:43
-
$\begingroup$ increase the size of bootstrap samples.. so that on get both the class in every sample. $\endgroup$ Nov 17, 2017 at 8:09
-
$\begingroup$ @ArpitSisodia that will still result in unbalanced samples. You would have to use sampling weights to oversample the rarer class in each bootstrap sample before constructing the tree. $\endgroup$ Apr 11, 2018 at 15:54
-
3$\begingroup$ This opinionated answer is misleading in that random forest is a great option, especially since an RF can easily be class weighted. Please follow the helpful best practice of providing a counter proposal when saying no, otherwise saying no is more harmful than it is helpful. $\endgroup$ Feb 9, 2020 at 1:37
-
$\begingroup$ @SwimBikeRun You're right about answering in general -- this was a long time ago! For that matter, I don't think this answer is even correct! $\endgroup$ Aug 17, 2021 at 18:48
-
$\begingroup$ shaowtalker: It seems decent to me. Can you edit the parts you think are incorrect? (Saying "RF can be great but only if you get the exact parameters right" is a two-edged sword) $\endgroup$– smciFeb 2, 2022 at 20:09
$\begingroup$
$\endgroup$
2
Unbalanced classes are only a an issue if you also have misclassification cost imbalance. If there are small minority classes and it is not more expensive to classify them as a majority class than the other way around, then the rational thing to do is to allow misclassification of minority classes.
So let's assume you have class and cost imbalance. There are multiple ways to deal with this. Max Kuhn's book "Applied predictive modeling" has a good overview in chapter 16. Those remedies include using a cutoff other than 0.5 which reflects the unequal costs. This is easy to do in binary classification as long as your classifier outputs label probabilities (trees and forests do this). I haven't looked into it for multiple classes yet. You can also oversample the minority class to give it more weight.
-
$\begingroup$ I don't think this is correct. If I have equal misclassification cost but my model is biased to overpredict one class, I am still left with a biased model at the end of the day. $\endgroup$ Oct 29, 2016 at 13:09
-
$\begingroup$ It wouldn't matter though. Cancer cases are much fewer than healthy patients. Yet you need to reliably predict the cancer patients because missing one is much more expensive than predicting one too many. If you had a data-set with 99.9% healthy people and 0.1% common cold cases, the best classifier would simply ignore those common cold cases. $\endgroup$ Oct 29, 2016 at 13:57