1
$\begingroup$

I am doing a multiple linear regression on a set of variables (observations over time). As commonly observed, my model underestimates high values and overestimates low values of the dependent variable $y$. This is clearly seen when I order $y$ (thus, lose the time structure), and plot it with the predicted values (ordered with respect to the ordering of $y$, not losing the "pairs"). In blue are the observed $y$ and in red is the predicted outcome.

enter image description here

The under/overestimation is clear, as the two newly fitted lines have different inclinations. A simple way to correct this bias would be to add to the slope of the red line, so that it equals the slope of the blue line:

enter image description here Plotting $y$ against predicted values before and after this bias correction looks like:

enter image description here

The correlation between $y$ and predicted values substantially increases.

I understand that adding independent variables in one's linear model, that do not correlate with your dependent variable, would cause a model to predict its mean - therefore over- and underestimating its low and high values, respectively. I also understand that this phenomenon has nothing to do with the behavior of the residuals, as pointed out in this post, for example. My residuals before correction look fine:

enter image description here

Also after correction (maybe a bit more correlated):

enter image description here

How can I interpret the need for this apparently linear bias, in my linear model? Is it fine to apply the same bias correction, when using this model for predictions? Any insights and clarifications are very welcome.

$\endgroup$
5
  • $\begingroup$ I don't get what you're doing with the red plot, but the blue dots indicate a quadratic or exponential relationship that your model misses. Either could be incorporated into a linear model, however e.g. $y=\beta_0 + \beta_1x + \beta_2x^2$ (which is a linear model, even if it doesn't look linear at first). $\endgroup$
    – Dave
    Jun 25, 2020 at 17:10
  • $\begingroup$ Which red plot do you mean? First, I just "lift" the too low modeled values and "dampen" the too high modeled values. In the second plot, I plot observations against model, before and after correction. Adding non-linear variables seems a good idea. But is it okay to do this bias correction? $\endgroup$
    – ouranos
    Jun 25, 2020 at 17:18
  • $\begingroup$ Are you using time as an independent variable? If so, is there a necessity to do so? It looks like you are modeling with observations time as a variable but you aren't using a time series model. $\endgroup$
    – Stephen G
    Jun 25, 2020 at 17:46
  • $\begingroup$ No, time is not a variable in the model. $\endgroup$
    – ouranos
    Jun 25, 2020 at 21:15
  • $\begingroup$ The lines in the first figure are just to show how the model is biased towards the mean of the observations. $\endgroup$
    – ouranos
    Jun 26, 2020 at 2:24

1 Answer 1

3
$\begingroup$

This apparent bias was a confusing way to put a symptom of a not perfectly fitted model. Every linear model, in which the coefficients are estimated by minimizing the sum of squared errors (the ERMS), shows the same.

If one tries to make a linear regression model using explanatory variables that are uncorrelated with the dependent variable, the variance of the predicted time series will be relatively small, and the regression coefficients will converge to zero, because the sum of squared variances is minimized (left). It is a direct consequence that the model will overestimate low values and underestimate high values. Ordering the dependent variable in ascending order is just another - and confusing - way to visualize the same thing (middle). Fitting yet another line to the red dots here and rotate them would destroy the regression model. One would have to scale each red dot by a different value, thus missing the point of having a couple of regression coefficients capable of scaling the explanatory variables as a whole.

enter image description here

If the explanatory variables are a bit more correlated, the same is observed. Model has lower variance than dependent variable, therefore model doesn't reach peaks, and overall over- and underestimates high and low values. This is when someone feels the urge to just rotate the red dots anti-clockwise, but it does not make any sense, if the goal is to have one regression model, and not as many scaling factors as data points.

enter image description here

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.