What does this plot tell me about my linear model?

Question

I have fit the following linear model, I tested the response by looking at a qq plot and it is almost perfectly linear. When i fit the model though, and study the predicted vs observed plot, It looks like this:

What does this tell me exactly? It seems to me that a better fitting line would be gained if i pivoted the model slightly to match the slope of the points. I'm not sure what I can do to gain a better predictive model.

edit

i trained the linear model on my training set. The 'predicted' in the plot are the result of applying that model to an independent 'test' set, and comparing it wth the observed values for that test set. The line is found by abline(0,1)

Answer 1

This sort of "problem" occurs quite naturally and can look this way without actually indicating a problem. (There might be some problem, but a pattern similar to this doesn't necessarily indicate one.)

It's a consequence of regression to the mean and arises directly out of fitting the conditional mean (i.e. it's exactly what you expect to see with regression).

One thing that might throw off some answerers is that you have your plot "backward" to what most of us are used to -- with the random variable on the x axis rather than the y-axis.

Here I have generated some data according to a regression model (with a normally distributed predictor and conditionally normal response) and fitted a model of the same form as the one that generated the data. Here's the corresponding plot to yours drawn the other way around:

Looking at the slice between the blue lines, the red line (which is just the line with slope 1 and intercept 0) passes very near the mean of the $y$ in that slice. That is, $\hat{y}\approx E(Y|x)$.

You are asking if you should "tweak" your line to lay closer to the major axis of the roughly elliptical point cloud ... but that is not going to be the "best fit" line, and will tend to overpredict the mean for large $y$ values and underpredict it for small y values.

If the regression assumptions are reasonable, and assuming you actually want to predict $E(Y|x)$, then there's nothing wrong here -- you see exactly what you should.

---

A case where you might see something like this, and where it might be an issue:

However, if your line at the edge of the cloud doesn't pass near the middle of small slices (vertical ones in my case) that might indicate that you have some underprediction (such as might occur if you're shrinking coefficients).

That may or may not be a problem: shrinking coefficients toward zero is often quite useful; that will lead to bias but bias isn't the whole story of fitting.

A small amount of bias toward zero (shrinkage) in the coefficients will produce a slightly "shallower" fit than the least squares line (on my plot; steeper on yours). That's not necessarily a problem at all.

It's only if the bias is larger than you want it to be that there would be any need to act at all. Otherwise it could still be doing exactly what it should.

So I don't see a problem here -- it looks to me like your model is doing what it should.

For reference, here's the plot from the question flipped around:

There's some hint that it's slightly biased toward 0 (which as mentioned, may not be a problem), and also perhaps a slight suggestion of a nonlinear relationship (which might potentially be a problem).

Answer 2

What these results tell you is something that is almost inevitable in this type of modeling: a model based on a training set will not fit a test set as well. The type of problem you face, with a difference in the slope of the relation between your linear predictor and the outcome variable between the training and test sets, seems to be one of calibration, as noted here in the context of logistic regression.

Your suggestion to "pivot" the slope of the line is similar to the general idea of "shrinking" regression coefficients to improve predictive ability on new data, but you would be better off using established methods like those provided by the rms package in R. Note that these efforts necessarily entail making a bias-variance tradeoff in predictive modeling. If you are unfamilar with that tradeoff, you should read An Introduction to Statistical Learning or a similar general reference.

Also, separate training and test sets might not be the most efficient way to use your data; developing the model on the entire data set and checking and adjusting calibration by bootstrap resampling may be better. Consider consulting Frank Harrell's course notes or his book for more detail.

---

Please check that your axes are labeled correctly. Typically one expects predicted values to be over-optimistic, with a wider range of predicted values than of observed. Also, the range of your results suggests that you might be using proportions as your outcome variable. If so, then there might be an issue if you used standard linear regression to develop your model.

Answer 3

I think most of the relevant information here has been provided by @EdM. Let me add a few thoughts:

Regarding your plot, I would put the predicted values on the x-axis and the observed values on the y-axis. That is the way scatterplots are more typically constructed and may help with interpretation. In addition, I would make the plot square and force the plotting area to range over the same possible values (say, $.1$ to $.9$) on both dimensions. Lastly, I would plot a LOWESS fit as well as the one to one line. These should make what is going on easier to see.

My guess is that your fitted model has too shallow a slope relative to the test data, but that the mean prediction is not biased (much). That shouldn't happen often. If your data were randomly split into train and test, they should be very similar, so the slopes wouldn't differ by much. This is also true given that you seem to have a lot of data. That should make estimates fairly stable. For me, these facts raise some questions:

• Was the split random, or was it by some pre-existing group structure (e.g., males vs. females)?
• Did you do a lot of fitting to get your model (such that it was overfitted)? More specifically, what was your modeling process?
• Do you have a lot of variables or complex functions of variables in the original model (and/or relative to the amount of data)?

More information is needed here.

It is also possible that you are missing some curvature in the data.