0
$\begingroup$

I would like to calculate confidence intervals for predicted probabilities of a class membership obtained with randomForest.

I know I could use predict.all in randomForest(), which gives me the predictions of all the trees and hence a way of calculating the variance of the average prediction. However, from reading the following paper by Wager, Hastie and Effron, the variance obtained this way is biased upwards, https://arxiv.org/pdf/1311.4555.pdf

This paper presents a bias-corrected estimator of variance and states on page 8 that it has been implemented in randomForest.

I have checked the documentation for randomForest but can't find any reference to it. https://www.rdocumentation.org/packages/randomForest/versions/4.6-14

Can anyone shed light on this?

Thanks in advance.

$\endgroup$
  • $\begingroup$ Hi, welcome to the site. To be clear, if your random forest predicts that the probability of an unlabelled example being equal to class 1 is 0.57, you're looking to say that it's, for example $0.57 \pm 0.03$ with 95% confidence? If so, that doesn't make sense/isn't defined. Confidence intervals are generally used to say things like "we are x% confident that a value lies in the range $[a,b]$. This either makes sense if you're trying to estimate a parameter (but random forests are non parametric) or if you're trying to estimate a continuous value (in regression), which you're not. $\endgroup$ – gazza89 Oct 26 at 14:11
  • $\begingroup$ Hi Gazza, thanks for taking the time to think about it. An RF is an ensemble model and gives the average vote or prediction from all the trees. The trees give different votes or predictions and so the CI reflects their distribution. It is a valid question. Check out arxiv.org/pdf/1311.4555.pdf $\endgroup$ – user2888990 Oct 28 at 22:51
  • $\begingroup$ I believe that the correct way to interpret the spread of probabilities predicted by the constituent trees in the random forest (given you use bootstrap sampling), is as follows: You have N training examples, which ultimately are sampled from some distribution. If you could go back to that distribution and take multiple samples of size N, and trained a decision tree for each, how would your predictions change. Thus in a sense, I guess with a large enough ensemble, you could think of your empirical the range which contains 95% of predictions as the 95% CI. $\endgroup$ – gazza89 Oct 28 at 23:38
  • $\begingroup$ That said, I'm not sure there's much meaning to a confidence interval on a probability. Uncertainty is already reflected in the probability. If you say there's a probability of 0.57 of an event happening, that reflects your uncertainty entirely. It makes sense when people say a coin has a probability of $0.57 \pm 0.03$ of coming up heads because p is a parameter and you're reflecting uncertainty on that parameters' true value. $\endgroup$ – gazza89 Oct 28 at 23:41
  • $\begingroup$ Thanks Gazza. The reference I mentioned looks at it as 1 - How much would predictions change if a different data set was used to train it? And, 2 - which predictions is the random forest more confident about? They have formulated a way of calculating the variance using the infinitesimal jacknife. This approach is used in the library(ranger) but you need to use a probability forest and not a classification forest. $\endgroup$ – user2888990 Nov 1 at 17:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.