I have found multiple questions here (e.g. this) and great academic papers (e.g. this and this) about calculating prediction intervals for Random Forest and other techniques applied to regression problems.
In the case of classification tasks, it is obvious that it does not make sense to consider prediction probabilities of predicted classes. This has been discussed before here, here and here in the context of binary prediction with logistic regression, and also here in the context of kNN regressions.
However, what about the predicted class probabilities? This comment suggests that as an option for prediction intervals for logistic regression, which, however, is not as easy to achieve theoretically as in the usual ways done for OLS, as pointed by this comment and this question.
Still, my intuition is that one could perhaps use non-parametric bootstrapping to estimate prediction intervals for the class probabilities - not only for logistic regression, but also Random Forests. So, my questions:
(1) would it be theoretically reasonable (that is, what are the limitations) to estimate prediction intervals for class probabilities in Random Forests classification tasks?
(2) how could that be done exactly, that is, would one simply re-sample with replacement from the training data and then re-train each time?
(3) if not, what other methods could be used?
Laterally, online or academic references about the possibility of estimating prediction intervals for class probabilities.
