# Probability extraction from random forest classifier

I have a random forest to perform classification. I need the real probability of the predicted class.

They take a feature vector $$X$$ and output a predicted class $$C$$. Additionally we can compute the confidence level $$L$$ over the predicted class. It is the percentage of trees that voted for that particular class, say 80% of the trees voted for class $$C1$$ then $$L$$ is 0.8.

• How can we relate the value $$L$$ to the real probability of $$C$$ given the feature vector $$X$$?
• Can we simply use $$L$$ as a reliable probability measurement?
• Are the classifier performance values (true/false positive rate, etc.) somehow considered into the value $$L$$, or do I need to use $$L$$ as a prior estimate to compute the real probability given the classifier performance parameters?

What you reference is an assessment of output calibration and calibrating outputs that are not calibrated.

By calibration, it is meant that a predicted value $$\hat p_i\in[0,1]$$ truly corresponds to an event probability of $$\hat p_i$$. That is, are the model outputs telling the truth?

The Scikit-learn documentation has a pretty good page on this topic.

How can we relate the value L or 'Probability' to the real probability of C given the feature vector X?

This is what the calibration process does. Two possible techniques are Platt scaling and isotonic regression, both of which are discussed at that Scikit-learn link.

Can we simply use L as a reliable probability measurement?

Maybe. Many machine learning models are overconfident with their predictions. For instance, in that Scikit-learn documentation, the random forest model is shown to lack good calibration and to be overconfident. If you run a function like calibration_curve in Scikit-learn or val.prob in the R package rms and find that your predicted probability values align with the reality of event occurrence, then you have evidence in support of using the predictions as they are, without a requirement to do a further step of calibrating those predictions.

Are the classifier performance values (true/false positive rate, etc) somehow considered into the value L, or do I need to use L as a prior estimate to compute the real probability given the classifier performance parameters?

The true "classification" performance metrics like accuracy, sensitivity, specificity, etc, do not relate to calibration. In fact, to calculate these, it is necessary to apply a decision rule on top of the raw model predictions, so these do not really refer to the same model; with calibration applying to the original model and accuracy etc. applying to the two-stage pipeline of original model > decision rule. Indeed, many decision rules can be used based on one set of original predictions. There is not even one "accuracy" value corresponding to a given model. That's how we wind up with ROC and PR curves, by varying to decision rule and seeing how the sensitivity/recall, specificity, and precision vary.