10
$\begingroup$

I am using the randomForest package in R (R version 2.13.1, randomForest version 4.6-2) for regression and noticed a significant bias in my results: the prediction error is dependent on the value of the response variable. High values are under-predicted and low values are over-predicted. At first I suspected this was a consequence of my data but the following simple example suggests that this is inherent to the random forest algorithm:

n = 1000; 
x1 = rnorm(n, mean = 0, sd = 1)
response = x1
predictors = data.frame(x1=x1) 
rf = randomForest(x=predictors, y=response)
error = response-predict(rf, predictors)
plot(x1, error)

I suspect the bias is dependent on the distribution of the response, for example, if x1 is uniformly-distributed, there is no bias; if x1 is exponentially distributed, the bias is one-sided. Essentially, the values of the response at the tails of a normal distribution are outliers. It is no surprise that a model would have difficulty predicting outliers. In the case of randomForest, a response value of extreme magnitude from the tail of a distribution is less likely to end up in a terminal leaf and its effect will be washed out in the ensemble average.

Note that I tried to capture this effect in a previous example, "RandomForest in R linear regression tails mtry". This was a bad example. If the bias in the above example is truly inherent to the algorithm, it follows that a bias correction could be formulated given the response distribution one is trying to predict, resulting in more accurate predictions.

Are tree-based methods, such as random forest, subject to response distribution bias? If so, is this previously known to the statistics community and how is it usually corrected (e.g. a second model that uses the residuals of the biased model as input)?

Correction of a response-dependent bias is difficult because, by nature, the response is not known. Unfortunately, the estimate/predicted response does not often share the same relationship to the bias.

$\endgroup$
1
4
$\begingroup$

It is perfectly as you suspect -- the fact that leaf nodes contain means over some set of objects make any regression tree model tighten the response distribution and make any extrapolation impossible. Ensemble of course does not help with that and in fact make situation worse.

The naive solution (and dangerous because of overfitting) is to wrap the model in some kind of classical regression which would rescale the response to its desired distribution.

The better solution is one of the model-in-leaf tree models, like for instance MOB in party package. The idea here is that partitioning of feature space should end when the problem is simplified not to a simple value (as in regular tree) but to a simple relation (say linear) between the response and some predictors. Such relation can be now resolved by fitting some simple model which won't disturb the distribution or trim extreme values and would be able to extrapolate.

$\endgroup$
2
  • $\begingroup$ Why would rescaling the response distribution lead to overfitting? The ordering relationship between any two of the forests outputs will retain the same order after pushing it through a polynomial fit to rescale it, so decision making from the model will not be impacted if you base your decision on quantiles of the output. $\endgroup$
    – Jase
    Feb 6 '14 at 19:05
  • $\begingroup$ You are right; I was thinking about a scenario when the RF produces totally nonsense answer and regression converts it into something with deceivingly better $R^2$. $\endgroup$
    – user88
    Feb 6 '14 at 21:23
3
$\begingroup$

I had exactly the same issue with Conditional RF accessed via the rattle package. I emailed Graham Williams (author of rattle) about it, who kindly forwarded my query to the cforest authors, who responded and suggested playing with two parameters that do not actually seem to be referenced anywhere in the CRF documentation, but which nonetheless seemed to address the problem, namely minplit = 2 and minbucket = 1.

$\endgroup$
1
  • $\begingroup$ minsplit, I presume you mean $\endgroup$
    – smci
    Jul 22 '15 at 4:33
2
$\begingroup$

You should be estimating the optimal value of mtry and sampsize by minimizing the out of sample "cross-validated error" over a grid of different mtry, sampsize parameters, for any corresponding response variable for a fixed set of features and then make any conclusions-in terms of the results. You may create a combination of the grid parameters using expand.grid.

$\endgroup$
1
  • 2
    $\begingroup$ This is a good suggestion for building models, in general optimized parameters will yield more accurate predictions. However, in this case the bias exists regardless of tuning, as far as I can tell. There is only one predictor so mtry must be 1. Any sample size less than n only serves to magnify the bias, and nodesize has little effect. $\endgroup$
    – rumbleB
    May 18 '12 at 18:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.