Is it valid to use the distribution of individual tree predictions in a Random Forest to approximate predictive uncertainty? Methods of estimating the uncertainty associated with a specific prediction from a Random Forest (Let's say Regression) by looking for a distribution to construct a CI from are quite involved/complex (jackknife and all that magic). I assume that since those methods exist, simply taking the individual predictions of trees and using that as a distribution is somehow invalid? Is this due to how the trees are constructed and the fact that they are not independent? I feel like I am missing something very obvious.
 A: No, this is probably not a good idea. The concept behind a RFR is to average a lot of "bad" predictions in order to get a good one. This does not necessarily entail that the individual predictions are realistic. This is something that you would need to construct a valid uncertainty range. For example, If you have many bad predictions that are too high, but equally many predictions that are too low (by a similar magnitude as the ones that tare too high), then they will cancel each other out, and the mean prediction can make sense. But if you use the individual predictions as distribution, then the distribution would be much too broad.
However, there is a technique called quantile regression forest that extends the concept of a RFR to probabilistic predictions. (https://www.jmlr.org/papers/volume7/meinshausen06a/meinshausen06a.pdf). In a RFR, each forest outputs an expectation value, and the individual expectation values are then averaged. The expectation value of a single tree is the mean value of all the training samples in the target leaf (the y-values of all training samples that end up in that leaf). In a QFR, each tree outputs a quantile (e.g. the 0.9 quantile), and this is computed as the quantile of all the training y-values in that leaf. The quantiles of the individual trees are then averaged (as a weighted average) and give a final quantile. If you predict different quantiles (e.g. 0.1 and 0.9) you can construct a quantile-range, which can be used as a measure of uncertainty of the prediction. This is different from what Asker is proposing. They propose to use the distribution of the expectations values of the single trees, but a QFR uses the distribution of y-values in the target leaf.
