# RandomForest , confidence interval for class membership probabilities

I would like to calculate confidence intervals for predicted probabilities of a class membership obtained with randomForest.

I know I could use predict.all in randomForest(), which gives me the predictions of all the trees and hence a way of calculating the variance of the average prediction. However, from reading the following paper by Wager, Hastie and Effron, the variance obtained this way is biased upwards, https://arxiv.org/pdf/1311.4555.pdf

This paper presents a bias-corrected estimator of variance and states on page 8 that it has been implemented in randomForest.

I have checked the documentation for randomForest but can't find any reference to it. https://www.rdocumentation.org/packages/randomForest/versions/4.6-14

Can anyone shed light on this?

• Hi, welcome to the site. To be clear, if your random forest predicts that the probability of an unlabelled example being equal to class 1 is 0.57, you're looking to say that it's, for example $0.57 \pm 0.03$ with 95% confidence? If so, that doesn't make sense/isn't defined. Confidence intervals are generally used to say things like "we are x% confident that a value lies in the range $[a,b]$. This either makes sense if you're trying to estimate a parameter (but random forests are non parametric) or if you're trying to estimate a continuous value (in regression), which you're not. – gazza89 Oct 26 at 14:11
• That said, I'm not sure there's much meaning to a confidence interval on a probability. Uncertainty is already reflected in the probability. If you say there's a probability of 0.57 of an event happening, that reflects your uncertainty entirely. It makes sense when people say a coin has a probability of $0.57 \pm 0.03$ of coming up heads because p is a parameter and you're reflecting uncertainty on that parameters' true value. – gazza89 Oct 28 at 23:41