I have been playing with the Kaggle House Prices dataset for sometime. I have been using only the non-categorical features.

After fitting LASSO I plotted the residual error vs true value scatter plot. Blue points are from training set and red points are from validation set.

True Value vs Residual Error for LASSO

As expected there are some outliers with high residual error. Most of the residuals are clustered around zero. I think it is expected that the mean value of residual errors should be zero.

But the more interesting question is how does the mean residual error change across the whole range of output. In this case the residuals tend to be positive when outputs are high and residuals tend to be negative when outputs are low. There seems to be a linear relationship. This relationship is present in both the train set and validation set. Apart from LASSO I also tried Ridge and OLS with similar results.

I also tried a random forest model. The result is almost same. There is more overfitting but the general trend is there. Even random forest fails to capture this trend.

So my question is why cannot neither a linear nor a non-linear model learn this linear relationship?

True Value vs Residual Error for Random Forest

  • $\begingroup$ Check point 3 in the link included. You have to standardize your residuals first, though. It speaks to Scale-Location plots and their use in identifying heteroscedasticity data.library.virginia.edu/diagnostic-plots $\endgroup$
    – ERT
    Commented Jul 12, 2018 at 16:44
  • $\begingroup$ @ERT One difference between the plot here and there is that on X axis I have the True value while there they have the predicted value. When plotted with predicted value in X axis this trend is actually not present. $\endgroup$ Commented Jul 12, 2018 at 18:05
  • $\begingroup$ A common question. This always happens $\endgroup$
    – Glen_b
    Commented Jul 13, 2018 at 9:22
  • $\begingroup$ stats.stackexchange.com/questions/96563/… $\endgroup$
    – Glen_b
    Commented Jul 13, 2018 at 9:24


Browse other questions tagged or ask your own question.