Linear bias in linear model?

Question

I am doing a multiple linear regression on a set of variables (observations over time). As commonly observed, my model underestimates high values and overestimates low values of the dependent variable $y$. This is clearly seen when I order $y$ (thus, lose the time structure), and plot it with the predicted values (ordered with respect to the ordering of $y$, not losing the "pairs"). In blue are the observed $y$ and in red is the predicted outcome.

The under/overestimation is clear, as the two newly fitted lines have different inclinations. A simple way to correct this bias would be to add to the slope of the red line, so that it equals the slope of the blue line:

Plotting $y$ against predicted values before and after this bias correction looks like:

The correlation between $y$ and predicted values substantially increases.

I understand that adding independent variables in one's linear model, that do not correlate with your dependent variable, would cause a model to predict its mean - therefore over- and underestimating its low and high values, respectively. I also understand that this phenomenon has nothing to do with the behavior of the residuals, as pointed out in this post, for example. My residuals before correction look fine:

Also after correction (maybe a bit more correlated):

How can I interpret the need for this apparently linear bias, in my linear model? Is it fine to apply the same bias correction, when using this model for predictions? Any insights and clarifications are very welcome.

Answer 1

This apparent bias was a confusing way to put a symptom of a not perfectly fitted model. Every linear model, in which the coefficients are estimated by minimizing the sum of squared errors (the ERMS), shows the same.

If one tries to make a linear regression model using explanatory variables that are uncorrelated with the dependent variable, the variance of the predicted time series will be relatively small, and the regression coefficients will converge to zero, because the sum of squared variances is minimized (left). It is a direct consequence that the model will overestimate low values and underestimate high values. Ordering the dependent variable in ascending order is just another - and confusing - way to visualize the same thing (middle). Fitting yet another line to the red dots here and rotate them would destroy the regression model. One would have to scale each red dot by a different value, thus missing the point of having a couple of regression coefficients capable of scaling the explanatory variables as a whole.

If the explanatory variables are a bit more correlated, the same is observed. Model has lower variance than dependent variable, therefore model doesn't reach peaks, and overall over- and underestimates high and low values. This is when someone feels the urge to just rotate the red dots anti-clockwise, but it does not make any sense, if the goal is to have one regression model, and not as many scaling factors as data points.