Random forest regression - residuals correlated with response

Question

I am trying to use Random Forest regression. I have a response variable:

y = rnorm(10000, mean=0, sd=3)

And a few predictor variables (which are just the response with added noise):

x = data.frame(v1=y + rnorm(10000, mean=0, sd=3), v2=y + rnorm(10000, mean=0, sd=3), v3=y + rnorm(10000, mean=0, sd=3))

I build the random forest:

r = randomForest(x, y)

The model is good, explaining ~73% of the variance. However, when I look at the residuals:

plot(y, y - r$predicted)

Instead of being centered around zero, they are correlated with the response variable. It seems that the model should correct this. Maybe, since each OOB prediction is an average, this behavior is some kind of "regression to the mean"? Does anyone know why this happens? Is anything I can do about it?

I am trying to build a model and use the residuals to estimate something. Right now, they are useless because they only reflect the value I'm trying to predict. If anyone can help, I'd really appreciate it!

Answer 1

This is exactly what you should expect to see.

You see exactly the same thing with an ordinary regression model; if the model has $R^2$ higher than 0, residuals are correlated with the response. It's the fitted values that the residuals are uncorrelated with.

Specifically, each regression captures all of the variation in $y$ that's explained by the predictors. Doing that means that $\hat{y}$ is uncorrelated with the residual -- because if it were not uncorrelated, there would be uncaptured variation that the $x$'s could capture. As a result of capturing everything that the $x$'s can capture in the regression model, there must be a correlation of residuals with the actual $y$ values:

$\text{Cov}(y,y-\hat{y}) = \text{Cov}(y-\hat{y},y-\hat{y})+ \text{Cov}(\hat{y},y-\hat{y})= \text{Var}(y-\hat{y})+0=\sigma^2$

-- since that covariance is positive, the correlation will be.

Regularization changes things somewhat but the same correlation appears.

[The component of $y$ that's uncaptured by the model, even if it's just pure noise, is obviously correlated with itself. That's really all we're seeing here]

Such a correlation will be present in every one of the regressions, and so correlation will carry over to the random forest as a whole.

There's nothing to fix here, it's actually supposed to do that. You see it even if you fit exactly the model that generated your data.

 x = rbeta(1000,1.8,1.8)*20+5     # make some x variable
 y = 0.8+0.3*x+rnorm(1000,0,1.5)  # simulate a regression model
 lmfit = lm(y~x)                  # fit the true regression model
 plot(y,lmfit$residual)           # voila!                             $
 cor(y,lmfit$residual)

$\qquad$

You'll see a correlation in the vicinity of 70% for this example.

In fact if you increase the variance of the noise term (other things being equal), the correlation is stronger. Try the above with more noise:

 y = 0.8+0.3*x+rnorm(1000,0,2.5)  # simulate a regression model

and see what happens!

It may be that you will want to ask another question explaining the original problem you were trying to solve, because your choice of how to go about it may not have been ideal (it's hard to tell from what is here).