I have been playing with the Kaggle House Prices dataset for sometime. I have been using only the non-categorical features.

After fitting LASSO I plotted the residual error vs true value scatter plot. Blue points are from training set and red points are from validation set.

True Value vs Residual Error for LASSO

As expected there are some outliers with high residual error. Most of the residuals are clustered around zero. I think it is expected that the mean value of residual errors should be zero.

But the more interesting question is how does the mean residual error change across the whole range of output. In this case the residuals tend to be positive when outputs are high and residuals tend to be negative when outputs are low. There seems to be a linear relationship. This relationship is present in both the train set and validation set. Apart from LASSO I also tried Ridge and OLS with similar results.

I also tried a random forest model. The result is almost same. There is more overfitting but the general trend is there. Even random forest fails to capture this trend.

So my question is why cannot neither a linear nor a non-linear model learn this linear relationship?

True Value vs Residual Error for Random Forest


marked as duplicate by Glen_b regression Jul 13 '18 at 9:24

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • $\begingroup$ Check point 3 in the link included. You have to standardize your residuals first, though. It speaks to Scale-Location plots and their use in identifying heteroscedasticity data.library.virginia.edu/diagnostic-plots $\endgroup$ – ERT Jul 12 '18 at 16:44
  • $\begingroup$ @ERT One difference between the plot here and there is that on X axis I have the True value while there they have the predicted value. When plotted with predicted value in X axis this trend is actually not present. $\endgroup$ – Ibraheem Moosa Jul 12 '18 at 18:05
  • $\begingroup$ A common question. This always happens $\endgroup$ – Glen_b Jul 13 '18 at 9:22
  • $\begingroup$ stats.stackexchange.com/questions/96563/… $\endgroup$ – Glen_b Jul 13 '18 at 9:24