9
$\begingroup$

I have a general question about asymmetric costs. In machine learning problems, there are times when the cost of a false positive is different from the cost of a false negative. Accordingly, models should be built differently to account for this asymmetry in costs.

How is this done for a random forest?

Some possible ways are:

  1. Changing the information gain calculated when considering different splits in a given branch of a decision tree to account for asymmetry
  2. Adjusting the threshold from 0.5 within each leaf when assigning the predicted label of a positive class in a given decision tree
  3. Adjusting the threshold from 0.5 within the collection of decision trees when "voting" on the predicted label for the random forest
  4. Using ROC curves and choosing a different threshold than what is typically chosen (typically, the threshold closest to the top-left corner of the ROC graph is chosen as the "ideal")

Which of these way(s) are implemented to account for asymmetric costs, in practice?

$\endgroup$
17
  • $\begingroup$ Random Forest Classifiers are generally build to node purity, so (2) is not an option. $\endgroup$
    – Scholar
    May 22, 2019 at 9:11
  • $\begingroup$ Shouldn't this be done though? I agree the code that random forest is built on generally does not do this, but you could conceivably customize the node purity calculation to take into account different weightings for a positive and negative class. $\endgroup$
    – mnmn
    May 22, 2019 at 14:16
  • $\begingroup$ How is that possible? Since the leaf nodes consist of only one class, the ratio of class labels is either 0 or undefined, so applying different weights does nothing. $\endgroup$
    – Scholar
    May 22, 2019 at 14:55
  • $\begingroup$ (2) You do not want specific threshold values between nodes, you really are after tuning how the tree is built not how the tree evaluates. (3) Voting thresholds would be quite bad too, since one most forests will collect probabilities from the trees rather than 0 or 1s. (1) is implemented pretty much everywhere (and works well), since adding weights to samples is how ada boost is performed. Never thought about (4) need to think on it a while. $\endgroup$
    – grochmal
    May 22, 2019 at 15:04
  • $\begingroup$ @grochmal Something to add regarding the points you've made for (3) and (1): First, (3) and (4) are equivalent and common practice. Second, I don't see any reason why Ada Boost is relevant as a justification for why (1) is common practice. $\endgroup$
    – Scholar
    May 22, 2019 at 15:28

1 Answer 1

3
$\begingroup$

Misclassification costs can often be dealt with through class weights, the same way as unbalanced classes can. This means that if the misclassification cost is higher for a class, elements of such a class will be more influent when making predictions.

For decision trees and random forests, this has been shown in this paper by Breiman, that I would say puts together points 1 and 2 of your question.

Indeed, Weighted Random Forest uses a weighted version of the Gini Coefficient in order to make the splits. This means that the Gini Coefficient will be maximum when the weighted sum of the elements of each class is equal (normally, Gini is maximum when elements are evenly distributed within the classes) (1). At the same time, this also means that the threshold that is used when considering the majority class of a node will not be 0.5, but it will come from the ratio of the class weights. Finally, this also applies for predictions (2), as the threshold will be modified by the weights.

Unfortunately, to this day I do not know any major statistical package using this method, as class weights are usually used for over/undersampling of the classes, which is much more specific to unbalanced classes.

Finally, using the ROC score is always advised when your classes and/or your costs are not balanced, so that you can tweak the thresholds to balance the results of your classifier.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.