Do the predictions of a Random Forest model have a prediction interval? If I run a randomForest model, I can then make predictions based on the model.  Is there a way to get a prediction interval of each of the predictions such that I know how "sure" the model is of its answer.  If this is possible is it simply based on the variability of the dependent variable for the whole model or will it have wider and narrower intervals depending on the particular decision tree that was followed for a particular prediction?   
 A: This is easy to solve with randomForest.
First let me deal with the regression task (assuming your forest has 1000 trees).
In the predict function, you have the option to return results from individual trees. This means that you will receive 1000 column output. We can take the average of the 1000 columns for each row - this is the regular output RF would have produced any way. Now to get the prediction interval lets say +/- 2 std. deviations all you need to do is, for each row, from the 1000 values calculate +/-2  std. deviations and make these your upper and lower bounds on your prediction.
Second, in the case of classification, remember that each tree outputs either 1 or 0 (by default)and the sum over all 1000 trees divied by 1000 gives the class probablity (in the case of binary classification). In order to put a prediction interval on the probability you need to modify the min. nodesize option (see randomForest  docuementation for the exact name of that option) once you set it a value >>1 then the individual trees will output numbers between 1 and 0. Now, from here on you can repeat the same process as described above for the regression task.
I hope that makes sense.
A: The problem of constructing prediction intervals for random forest predictions has been addressed in the following paper:
Zhang, Haozhe, Joshua Zimmerman, Dan Nettleton, and Daniel J. Nordman. "Random Forest Prediction Intervals." The American Statistician,2019. 
The R package "rfinterval" is its implementation available at CRAN.
Installation
To install the R package rfinterval:
#install.packages("devtools")
#devtools::install_github(repo="haozhestat/rfinterval")
install.packages("rfinterval")
library(rfinterval)
?rfinterval

Usage
Quickstart:
train_data <- sim_data(n = 1000, p = 10)
test_data <- sim_data(n = 1000, p = 10)

output <- rfinterval(y~., train_data = train_data, test_data = test_data,
                     method = c("oob", "split-conformal", "quantreg"),
                     symmetry = TRUE,alpha = 0.1)

### print the marginal coverage of OOB prediction interval
mean(output$oob_interval$lo < test_data$y & output$oob_interval$up > test_data$y)

### print the marginal coverage of Split-conformal prediction interval
mean(output$sc_interval$lo < test_data$y & output$sc_interval$up > test_data$y)

### print the marginal coverage of Quantile regression forest prediction interval
mean(output$quantreg_interval$lo < test_data$y & output$quantreg_interval$up > test_data$y)

Data example:
oob_interval <- rfinterval(pm2.5 ~ .,
                            train_data = BeijingPM25[1:1000, ],
                            test_data = BeijingPM25[1001:2000, ],
                            method = "oob",
                            symmetry = TRUE,
                            alpha = 0.1)
str(oob_interval)

A: If you use R you can easily produce prediction intervals for the predictions of a random forests regression: Just use the package quantregForest (available at CRAN) and read the paper by N. Meinshausen on how conditional quantiles can be inferred with quantile regression forests and how they can be used to build prediction intervals. Very informative even if you don't work with R!
A: I've tried some options (this all WIP): 


*

*I actually made the dependent variable a classification problem with the results as ranges, instead of a single value. The results I got were poor, compared to using a plain value. I gave up this approach.

*I then converted it to multiple classification problems, each of which was a lower-bound for the range (the result of the model being whether it would cross the lower bound or not) and then ran all the models (~20), and then combined the result to get a final answer as a range. This works better than 1 above but not as good as I need it to. I'm still working to improve this approach.
I used OOB and leave-one-out estimates to decide how good/bad my models are.
