5
$\begingroup$

I've trained a random forest on a data set where the targets are in [0, 100]. I use a 5 fold cross validation framework to find the optimum mtry and then train the model on the whole data set using that mtry. When I make predictions on the training set I find that the low targets are over-predicted and the high targets are under-predicted and I don't know why this is happening. The RMSE would be lower if the predictions were just pushed farther from the mean value.Observed vs Predicted

Can anyone indicate why this is happening? When the model makes predicts on a holdout set this problem is exacerbated but I though it instructive to show the predictions back on the training set in this example. I could augment the model to correct for this and that would improve results on the holdout set, but I would like to understand why this is happening so that I might fix it without augmenting the model. The distribution of target values is given below:enter image description here

$\endgroup$
6
  • 1
    $\begingroup$ It would help if you posted a reproducible example. Thanks! $\endgroup$
    – Zach
    Nov 1 '13 at 14:05
  • $\begingroup$ Sorry, first post on this site. Are you suggesting I upload my data and the trained model (I'm using caret)? What is the best way to do this? Thanks. $\endgroup$ Nov 1 '13 at 14:31
  • $\begingroup$ something like this: model <- train(Sepal.Width~., data=iris); p <- predict(model, newdata=iris); plot(p, iris$Sepal.Width); lines(0:10, 0:10, col='red') $\endgroup$
    – Zach
    Nov 1 '13 at 18:30
  • $\begingroup$ Ok, jbowman described the cause of this behaviour. I'll include an example next time I ask a question. $\endgroup$ Nov 1 '13 at 21:52
  • $\begingroup$ Sounds good! I've found just generating some sample data that reproduces the problem often helps me understand the problem. $\endgroup$
    – Zach
    Nov 1 '13 at 22:55
6
$\begingroup$

This is actually to be expected, not just with random forests, and comes about as a consequence of the fact that the variance of the target variable = the variance of the model (the estimates) + the variance of the residuals (for least-squares type fitting procedures.) Given that the latter is positive, unless your model fits perfectly, it must be that the variance of the model < the variance of the target variable. As a result, the prediction vs. actual plot can't lie on the 45-degree line passing through 0; if it did, the variance of the target variable would be equal to the variance of the model, and there would be no room left for residual variance.

Here are four plots to illustrate this point with linear regression. In the first one, the error variance is relatively high, and, as a consequence, the predicted - vs - actual plot isn't anywhere near the diagonal line. In the second through fourth, the error variance is much lower, and the predicted - vs - actual plot gets much closer to the diagonal line.

First, the code:

x <- rnorm(1000)
y <- x + rnorm(1000,0,2) # rnorm(1000,0,1), rnorm(1000,0,0.5), rnorm(1000,0,0.1) 

plotlim <- range(y)
plot(predict(lm(y~x))~y,ylim=plotlim,xlim=plotlim)
abline(c(0,1))

Now, the plots:

enter image description here enter image description here enter image description here enter image description here

Consequently, there's no need to alter your fitting procedure or augment your model.

Further heuristic explanation: Note that this comes about because $\sigma^2_Y > \sigma^2_X$, in this particular linear regression model. Therefore, even with the true parameter values (in this case, 0 intercept and 1 slope), the plot of $Y$ will be more spread out than the plot of $X$, and, since the estimated values of $Y$ with the true parameter values will equal $X$, it will also be the case that the plot of $Y$ will be more spread out than the plot of the estimated values of $Y$. As a result, the estimated values vs. true values plot will not lie on a 45-degree line.

$\endgroup$
4
  • $\begingroup$ Thank you for the clear explanation. It makes sense to me now. Does this mean that the more you see this effect with a trained model, the more noise (your residual variance) there is in the data? $\endgroup$ Nov 1 '13 at 17:07
  • $\begingroup$ Yes, it does. The worst, of course, is when your best model fit is just a constant, i.e., all coefficients / other terms are effectively zero, in which case the residual variance = the target variable variance and the blob of points is a horizontal line. (I should have thought to put that into the plots.) $\endgroup$
    – jbowman
    Nov 1 '13 at 23:24
  • 1
    $\begingroup$ Nice answer jbowman, you should also note that Random Forests works by menas of averaging the prediction of many diverse models (CART). Think the jensen inequality: you have the MSE which can be decomposed in its variance and bias parts. A Random Forest always inherits the bias of its trees. It improves accuracy by reducing variance. People are quite worried about reducing bias in Random Forest you can google it. The R implementation actually has its own bias correction but I neither know how it works nor have I seen a substantial improvement when I have used it. $\endgroup$
    – JEquihua
    Nov 3 '13 at 17:23
  • $\begingroup$ @jbowman only 8 years late to the party and I've stumbled across this issue, so thanks for the helpful explanation. I wonder, is it reasonable to use a pipeline something like this: run the random forest, then do something like lm(prediction ~ truth), and use the result of the lm as an extra step to correct for such errors? Or is this really a bad idea? $\endgroup$
    – Mooks
    Apr 19 at 15:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.