# Predictions of Random Forest on training data don't lie around x=y line

I've trained a random forest on a data set where the targets are in [0, 100]. I use a 5 fold cross validation framework to find the optimum mtry and then train the model on the whole data set using that mtry. When I make predictions on the training set I find that the low targets are over-predicted and the high targets are under-predicted and I don't know why this is happening. The RMSE would be lower if the predictions were just pushed farther from the mean value.

Can anyone indicate why this is happening? When the model makes predicts on a holdout set this problem is exacerbated but I though it instructive to show the predictions back on the training set in this example. I could augment the model to correct for this and that would improve results on the holdout set, but I would like to understand why this is happening so that I might fix it without augmenting the model. The distribution of target values is given below:

• It would help if you posted a reproducible example. Thanks!
– Zach
Nov 1, 2013 at 14:05
• Sorry, first post on this site. Are you suggesting I upload my data and the trained model (I'm using caret)? What is the best way to do this? Thanks. Nov 1, 2013 at 14:31
• something like this: model <- train(Sepal.Width~., data=iris); p <- predict(model, newdata=iris); plot(p, iris$Sepal.Width); lines(0:10, 0:10, col='red') – Zach Nov 1, 2013 at 18:30 • Ok, jbowman described the cause of this behaviour. I'll include an example next time I ask a question. Nov 1, 2013 at 21:52 • Sounds good! I've found just generating some sample data that reproduces the problem often helps me understand the problem. – Zach Nov 1, 2013 at 22:55 ## 1 Answer This is actually to be expected, not just with random forests, and comes about as a consequence of the fact that the variance of the target variable = the variance of the model (the estimates) + the variance of the residuals (for least-squares type fitting procedures.) Given that the latter is positive, unless your model fits perfectly, it must be that the variance of the model < the variance of the target variable. As a result, the prediction vs. actual plot can't lie on the 45-degree line passing through 0; if it did, the variance of the target variable would be equal to the variance of the model, and there would be no room left for residual variance. Here are four plots to illustrate this point with linear regression. In the first one, the error variance is relatively high, and, as a consequence, the predicted - vs - actual plot isn't anywhere near the diagonal line. In the second through fourth, the error variance is much lower, and the predicted - vs - actual plot gets much closer to the diagonal line. First, the code: x <- rnorm(1000) y <- x + rnorm(1000,0,2) # rnorm(1000,0,1), rnorm(1000,0,0.5), rnorm(1000,0,0.1) plotlim <- range(y) plot(predict(lm(y~x))~y,ylim=plotlim,xlim=plotlim) abline(c(0,1))  Now, the plots: Consequently, there's no need to alter your fitting procedure or augment your model. Further heuristic explanation: Note that this comes about because$\sigma^2_Y > \sigma^2_X$, in this particular linear regression model. Therefore, even with the true parameter values (in this case, 0 intercept and 1 slope), the plot of$Y$will be more spread out than the plot of$X$, and, since the estimated values of$Y$with the true parameter values will equal$X$, it will also be the case that the plot of$Y$will be more spread out than the plot of the estimated values of$Y\$. As a result, the estimated values vs. true values plot will not lie on a 45-degree line.

• Thank you for the clear explanation. It makes sense to me now. Does this mean that the more you see this effect with a trained model, the more noise (your residual variance) there is in the data? Nov 1, 2013 at 17:07
• Yes, it does. The worst, of course, is when your best model fit is just a constant, i.e., all coefficients / other terms are effectively zero, in which case the residual variance = the target variable variance and the blob of points is a horizontal line. (I should have thought to put that into the plots.) Nov 1, 2013 at 23:24
• Nice answer jbowman, you should also note that Random Forests works by menas of averaging the prediction of many diverse models (CART). Think the jensen inequality: you have the MSE which can be decomposed in its variance and bias parts. A Random Forest always inherits the bias of its trees. It improves accuracy by reducing variance. People are quite worried about reducing bias in Random Forest you can google it. The R implementation actually has its own bias correction but I neither know how it works nor have I seen a substantial improvement when I have used it. Nov 3, 2013 at 17:23
• @jbowman only 8 years late to the party and I've stumbled across this issue, so thanks for the helpful explanation. I wonder, is it reasonable to use a pipeline something like this: run the random forest, then do something like lm(prediction ~ truth), and use the result of the lm as an extra step to correct for such errors? Or is this really a bad idea? Apr 19, 2021 at 15:46
• @Mooks, that sounds like a good new question (linking to this one) Dec 15, 2021 at 15:32