Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I think this is a straightforward question, although the reasoning behind why or why not may not be. The reason I ask is that I have recently written my own implementation of a RF and although it performs well it is not performing quite as well as I had expected (based on the Kaggle Photo Quality Prediction competition data set, the winning scores and some of the subsequent info that came available about what techniques were used).

The first thing I do in such circumstances is plot prediction error for my model, so for each given prediction value I determine the mean bias (or deviation) away from the correct target value. For my RF I got this plot:

Prediction value versus bias from correct target value

I'm wondering if this is a commonly observed bias pattern for RF (if not then it could perhaps be something specific to the data set and/or my implementation). I can of course use this plot to improve predictions by using it to compensate for the bias, but I'm wondering if there's a more fundamental error or shortcoming in the RF model itself that needs addressing. Thankyou.


My initial investigation is at this blog entry Random Forest Bias - Update

share|improve this question
It might be a feature of your data; have you tried running other RF implementation on the same dataset to see if it reproduces this effect? –  mbq Jan 23 '12 at 13:18
add comment

2 Answers

up vote 2 down vote accepted

(I'm far from expert. These are just musings from a junior statistician who has dealt with different, but loosely analogous, issues. My answer might be out of context.)

Given a new sample to be predicted, and an oracle which has access to a much larger training set, then maybe the "best" and most honest prediction is to say "I predict with 60% probability that this belongs in the Red class rather than the Blue class".

I'll give a more concrete example. Imagine that, in our very large training set, there is a large set of samples that are very similar to our new sample. Of these, 60% are blue and 40% are red. And there appears to be nothing to distinguish the Blues from the Red. In such a case, it's obvious that a 60%/40% is the only prediction a sane person can make.

Of course, we don't have such an oracle, instead we have lots of trees. Simple decision trees are incapable of making these 60%/40% predictions and hence each tree will make a discrete prediction (Red or Blue, nothing in between). As this new sample falls just on the Red side of the decision surface, you will find that almost all of the trees predict Red rather than Blue. Each tree pretends to be more certain than it is and it starts a stampede towards a biased prediction.

The problem is that we tend to misinterpret the decision from a single tree. When a single tree puts a node in the Red class, we should not interpret that as a 100%/0% prediction from the tree. (I'm not just saying that we 'know' that it's probably a bad prediction. I'm saying something stronger, i.e. that we should be careful we interpret as being the tree's prediction). I can't concisely expand on how to fix this. But it is possible to borrow ideas from areas of statistics about how to construct more 'fuzzy' splits within a tree in order to encourage a single tree to be more honest about its uncertainty. Then, it should be possible to meaningfully average the predictions from a forest of trees.

I hope this helps a little. If not, I hope to learn from any responses.

share|improve this answer
Fuzzy splits, got it, in the spirit of extreme RF (but possibly not so extreme?). I will try this as your explanation makes sense to me. Thanks. –  locster Jan 23 '12 at 23:55
[Random Forest - Thoughts on Bias Problem ](the-locster.livejournal.com/134241.html) "The key then (I think) is to use a non-uniform randomisation [of teh split threshold] such that the set of all split points when combined will recreate y = f(x), and approach a perfect representation of y = f(x) as the number of DTs in the RF tends towards infinity." –  locster Aug 23 '12 at 23:36
add comment

Yes. Most trees have a bias in the tails. See:

How should decision tree splits be implemented when predicting continuous variables?

"One potential issue with trees is that they tend to fit poorly in the tails. Think of a terminal node that captures the low range of the training set. It will predict using the mean of those training set points, which will always under-predict the outcome (since it is the mean)."

share|improve this answer
I dont think that comment applies to random forests –  Zach Jan 23 '12 at 2:28
I believe the reference implementation of random forests stop with ~5 observations in the nodes when the response variable is continuous. That would still introduce a small amount of bias if the splitting variable were continuous as well. Similar to why LOESS often looks better than a centered moving average at the edges of the support... –  Shea Parkes Jan 23 '12 at 14:35
add comment

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.