Random Forest Classifier Design

Question

When we use random forest for classification it constructs multiple trees and finally averages the output, in the case of classification it takes a majority vote among all the trees and assigns a class. In the case of predicting with random forest, the features are again passed down each tree and a majority vote is taken at the terminal nodes.

If we request for class probabilities, these probabilities are calculated by counting the proportion of each of the classes for all the trees, ex. for binary classification, $P(1|X)=\frac{\#1's}{\#1's+\#0's}$.

My Question:

Is it appropriate to calculate these probabilities by counting the votes and then designing a classifier, ie. computing AUC, misclassification error as a function of probability threshold, positive prediction negative prediction...

The logic seems circular, where we first classify by majority vote, then we count to get probabilities and then apply a probability threshold.

Can we do this?

Answer 1

You're right, but because this is generally not the best thing to do:

in the case of classification it takes a majority vote among all the trees and assigns a class.

That is a common conception about Random Forests, but it is not how many standard implementations work. Instead, the trees "vote" probabilistically: the individual trees assign probabilities for each record which are the ratio of the number of positive classes to total training examples in the terminal node containing the data point. Then these individual tree probabilities are averaged across all the trees to get an overall predicted probability.

Assigning class membership is no-where required, and is external to the random forest (an almost all other) machine learning algorithms.

Answer 2

I don't see how this is circular. RF predictions are the fraction of votes for a particular class. Those predictions can be ranked, and ROC analysis and AUC only cares about ranks of negatives and positives.

By contrast, if you're solely making predictions by majority vote, you're implicitly making a number of assumptions which are not necessarily appropriate to whatever problem you're trying to solve. For example, you can have AUC = 1 if all positives are scored at 0.49 and all negatives at 0.48. For another, if you pay a very large cost for FPs, then you should select an operating point at a low FPR.

Essentially every machine learning model (logistic regression, etc) are not classifiers, but yield some sort of score which, with a decision rule, can make binary decisions.