I am doing a project that involves fitting some observations to a model using ordinary least squares method. I have thousands of variables and millions of observables.

When I plot the True versus Predicted Values of the Observables, I obtain a plot that does not align with the x=y line. The line of best-fit details are also provided.

enter image description here

The residual vs Predicted value plot is here, with color showing the density (the data accumulates around 0)

enter image description here

I am not sure why this happens? Is it a problem with my model being too simple, or the data accumulation, and what can I research for improvements? Are there regression methods for fixing this?


1 Answer 1


Let's agree on notation: $Y$ is a target variable, $Y'$ is a fitted model output. Then, $Y - Y'$ is a residual term.

Indeed one expects to see on the plot of target vs. fitted to align with the line of the form

$$Y' = Y$$

The spread of data points around this line indicates how much information was not accounted for in the model. So, your expectation is correct. I am sorry I have made misleading comments on this point.

The summary looks OK.

You got a significantly large slope, and a non significant intercept. So the intercept is zero.

R^2 is high enough, and it is definitely more than zero in statistical sense.

$$Y = 0.623 * X$$ is effectively your model.

To your charts:

  1. My advice is to plot these data as a heatmap, and zoom-in on an area with the highest density. Then you should observe the dense cloud which forms a majority of your data goes in line with the diagonal. If you dont's see that that means your model does not have enough independent variables / chosen IV are weakly associated with the target.

  2. Residuals vs. fitted. It should not have a trend, nor linear nor nonlinear, so that variation of $Y - Y'$ is equal for all $Y'$. I could say your plot looks OK.

I don't say you could improve the fit by weithing cases. That, in turn, could change the shape of the tails in distribution, but make it worse in the middle. Better try different candidate independent variables, if you have a choice, of course.

What else to look at:

A prob. density for the residuals. Ideally, but not necessary, they should be close to a normal shape. Normally distributed residuals tell us that the model contains all the infomative regressors about the target, leaving so called "measurements errors" only, which are expected to be normally distributed.

Finally, this is a possible duplicate of: Interpret regression model actual vs predicted plot far off of y=x line and Interpret if residuals are "close enough" to a normal distribution

  • 1
    $\begingroup$ Thank you for your answer. Can you explain a bit more about the bit "your model output is not biased along the y axis"? Does it mean my slope is skewed because my model output is biased along the y axis? If possible, can you suggest ways to improve my model? Would any methods such as weighted least squares, or even generalized least squares be useful? $\endgroup$ Jan 30, 2021 at 12:52
  • $\begingroup$ That means intercept = 0. You got this almost perfectly. + your residuals should not depend on predicted, add a smoothing operator to the second chart to see that is true. $\endgroup$ Jan 30, 2021 at 14:04
  • $\begingroup$ I am still a bit unsure. From what you said do you mean my prediction model is good? I can see from analysing the results, that my model predicts better when the true is close to the mean value (0), but not as good for larger magnitude values. Can you suggest what I can try to improve my model? $\endgroup$ Jan 31, 2021 at 0:52
  • $\begingroup$ I will modify my naswer not to misguide you, and to give more advices. $\endgroup$ Jan 31, 2021 at 8:46
  • 1
    $\begingroup$ Thank you for your updated answer and advice. I followed you instruction to plot the density map of the True vs Predicted plot, and I can see that where the points are the densest, the data points are very close to the diagonal line with slope 1. Thank you for your help! $\endgroup$ Feb 1, 2021 at 1:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.