# Do Random Forests exhibit prediction bias?

I think this is a straightforward question, although the reasoning behind why or why not may not be. The reason I ask is that I have recently written my own implementation of a RF and although it performs well it is not performing quite as well as I had expected (based on the Kaggle Photo Quality Prediction competition data set, the winning scores and some of the subsequent info that came available about what techniques were used).

The first thing I do in such circumstances is plot prediction error for my model, so for each given prediction value I determine the mean bias (or deviation) away from the correct target value. For my RF I got this plot:

I'm wondering if this is a commonly observed bias pattern for RF (if not then it could perhaps be something specific to the data set and/or my implementation). I can of course use this plot to improve predictions by using it to compensate for the bias, but I'm wondering if there's a more fundamental error or shortcoming in the RF model itself that needs addressing. Thankyou.

My initial investigation is at this blog entry Random Forest Bias - Update

• It might be a feature of your data; have you tried running other RF implementation on the same dataset to see if it reproduces this effect?
– user88
Jan 23, 2012 at 13:18

(I'm far from expert. These are just musings from a junior statistician who has dealt with different, but loosely analogous, issues. My answer might be out of context.)

Given a new sample to be predicted, and an oracle which has access to a much larger training set, then maybe the "best" and most honest prediction is to say "I predict with 60% probability that this belongs in the Red class rather than the Blue class".

I'll give a more concrete example. Imagine that, in our very large training set, there is a large set of samples that are very similar to our new sample. Of these, 60% are blue and 40% are red. And there appears to be nothing to distinguish the Blues from the Red. In such a case, it's obvious that a 60%/40% is the only prediction a sane person can make.

Of course, we don't have such an oracle, instead we have lots of trees. Simple decision trees are incapable of making these 60%/40% predictions and hence each tree will make a discrete prediction (Red or Blue, nothing in between). As this new sample falls just on the Red side of the decision surface, you will find that almost all of the trees predict Red rather than Blue. Each tree pretends to be more certain than it is and it starts a stampede towards a biased prediction.

The problem is that we tend to misinterpret the decision from a single tree. When a single tree puts a node in the Red class, we should not interpret that as a 100%/0% prediction from the tree. (I'm not just saying that we 'know' that it's probably a bad prediction. I'm saying something stronger, i.e. that we should be careful we interpret as being the tree's prediction). I can't concisely expand on how to fix this. But it is possible to borrow ideas from areas of statistics about how to construct more 'fuzzy' splits within a tree in order to encourage a single tree to be more honest about its uncertainty. Then, it should be possible to meaningfully average the predictions from a forest of trees.

I hope this helps a little. If not, I hope to learn from any responses.

• Fuzzy splits, got it, in the spirit of extreme RF (but possibly not so extreme?). I will try this as your explanation makes sense to me. Thanks. Jan 23, 2012 at 23:55
• [Random Forest - Thoughts on Bias Problem ](the-locster.livejournal.com/134241.html) "The key then (I think) is to use a non-uniform randomisation [of teh split threshold] such that the set of all split points when combined will recreate y = f(x), and approach a perfect representation of y = f(x) as the number of DTs in the RF tends towards infinity." Aug 23, 2012 at 23:36
• Would the 60/40% predictions not be handled by a regression tree? The confidence would the class ratio in a leaf's partition (for the training set). Perhaps this can/has been expanded to deal with statistical power too Aug 3, 2017 at 0:07

Yes. Most trees have a bias in the tails. See:

How should decision tree splits be implemented when predicting continuous variables?

"One potential issue with trees is that they tend to fit poorly in the tails. Think of a terminal node that captures the low range of the training set. It will predict using the mean of those training set points, which will always under-predict the outcome (since it is the mean)."

• I dont think that comment applies to random forests
– Zach
Jan 23, 2012 at 2:28
• I believe the reference implementation of random forests stop with ~5 observations in the nodes when the response variable is continuous. That would still introduce a small amount of bias if the splitting variable were continuous as well. Similar to why LOESS often looks better than a centered moving average at the edges of the support... Jan 23, 2012 at 14:35