1
$\begingroup$

Methods of estimating the uncertainty associated with a specific prediction from a Random Forest (Let's say Regression) by looking for a distribution to construct a CI from are quite involved/complex (jackknife and all that magic). I assume that since those methods exist, simply taking the individual predictions of trees and using that as a distribution is somehow invalid? Is this due to how the trees are constructed and the fact that they are not independent? I feel like I am missing something very obvious.

$\endgroup$
  • $\begingroup$ Question is not so clear to me. What does CI stand for? The bottom line is that you want some method to estimate the uncertainty of the prediction of RFR, correct? $\endgroup$ – Saleh Oct 5 '20 at 15:41
  • $\begingroup$ confidence interval. To the second part, yes but not what I am asking. I am aware there are methods/papers on estimating uncertainty in the prediction of RFR/RFC, however I am inquiring as to the validity of the approach outlined in the question. $\endgroup$ – Alexis Drakopoulos Oct 6 '20 at 9:49
1
$\begingroup$

No, this is probably not a good idea. The concept behind a RFR is to average a lot of "bad" predictions in order to get a good one. This does not necessarily entail that the individual predictions are realistic. This is something that you would need to construct a valid uncertainty range. For example, If you have many bad predictions that are too high, but equally many predictions that are too low (by a similar magnitude as the ones that tare too high), then they will cancel each other out, and the mean prediction can make sense. But if you use the individual predictions as distribution, then the distribution would be much too broad.

However, there is a technique called quantile regression forest that extends the concept of a RFR to probabilistic predictions. (https://www.jmlr.org/papers/volume7/meinshausen06a/meinshausen06a.pdf). In a RFR, each forest outputs an expectation value, and the individual expectation values are then averaged. The expectation value of a single tree is the mean value of all the training samples in the target leaf (the y-values of all training samples that end up in that leaf). In a QFR, each tree outputs a quantile (e.g. the 0.9 quantile), and this is computed as the quantile of all the training y-values in that leaf. The quantiles of the individual trees are then averaged (as a weighted average) and give a final quantile. If you predict different quantiles (e.g. 0.1 and 0.9) you can construct a quantile-range, which can be used as a measure of uncertainty of the prediction. This is different from what Asker is proposing. They propose to use the distribution of the expectations values of the single trees, but a QFR uses the distribution of y-values in the target leaf.

$\endgroup$
  • $\begingroup$ Instead of looking at individual trees' predictions; what about looking at sampled groups of m<<n individual trees for a more robust notion of uncertainty? $\endgroup$ – xrdty Dec 11 '20 at 16:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.