6
$\begingroup$

I have fit the following linear model, I tested the response by looking at a qq plot and it is almost perfectly linear. When i fit the model though, and study the predicted vs observed plot, It looks like this:

enter image description here

What does this tell me exactly? It seems to me that a better fitting line would be gained if i pivoted the model slightly to match the slope of the points. I'm not sure what I can do to gain a better predictive model.

edit

i trained the linear model on my training set. The 'predicted' in the plot are the result of applying that model to an independent 'test' set, and comparing it wth the observed values for that test set. The line is found by abline(0,1)

| cite | improve this question | |
$\endgroup$
  • 1
    $\begingroup$ Why are you using observed vs predicted rather than standard predicted vs residuals (standard plot(fit) method)..? Moreover, are you sure that the red line is plotted correctly? It looks like you switched axes.. $\endgroup$ – Tim♦ Jul 27 '16 at 11:57
  • $\begingroup$ @Tim I did abline(0,1) isn't that correct? And it seemed intuitive to me to see how far my observations were from my predictions, these predictions are based on a test set, so the model wasn't trained on them, therefore they are not the 'fits' $\endgroup$ – dimebucker91 Jul 27 '16 at 12:08
  • $\begingroup$ To make sure we understand the question: is it correct that you developed your model on a test set, and that what you display in the graph of Predicted versus Observed is based on a separate validation set? If so, please edit your question so that is clear there. Clarifications in comments may have a chance of being lost. $\endgroup$ – EdM Jul 27 '16 at 13:49
  • $\begingroup$ @EdM please see the edit $\endgroup$ – dimebucker91 Jul 27 '16 at 14:12
  • $\begingroup$ How did you decide which cases are in the training or test set? Which type of model is it? Mixed effects? $\endgroup$ – user83346 Jul 27 '16 at 15:16
9
$\begingroup$

This sort of "problem" occurs quite naturally and can look this way without actually indicating a problem. (There might be some problem, but a pattern similar to this doesn't necessarily indicate one.)

It's a consequence of regression to the mean and arises directly out of fitting the conditional mean (i.e. it's exactly what you expect to see with regression).

One thing that might throw off some answerers is that you have your plot "backward" to what most of us are used to -- with the random variable on the x axis rather than the y-axis.

Here I have generated some data according to a regression model (with a normally distributed predictor and conditionally normal response) and fitted a model of the same form as the one that generated the data. Here's the corresponding plot to yours drawn the other way around:

enter image description here

Looking at the slice between the blue lines, the red line (which is just the line with slope 1 and intercept 0) passes very near the mean of the $y$ in that slice. That is, $\hat{y}\approx E(Y|x)$.

You are asking if you should "tweak" your line to lay closer to the major axis of the roughly elliptical point cloud ... but that is not going to be the "best fit" line, and will tend to overpredict the mean for large $y$ values and underpredict it for small y values.

enter image description here

If the regression assumptions are reasonable, and assuming you actually want to predict $E(Y|x)$, then there's nothing wrong here -- you see exactly what you should.

A case where you might see something like this, and where it might be an issue:

However, if your line at the edge of the cloud doesn't pass near the middle of small slices (vertical ones in my case) that might indicate that you have some underprediction (such as might occur if you're shrinking coefficients).

That may or may not be a problem: shrinking coefficients toward zero is often quite useful; that will lead to bias but bias isn't the whole story of fitting.

A small amount of bias toward zero (shrinkage) in the coefficients will produce a slightly "shallower" fit than the least squares line (on my plot; steeper on yours). That's not necessarily a problem at all.

It's only if the bias is larger than you want it to be that there would be any need to act at all. Otherwise it could still be doing exactly what it should.

So I don't see a problem here -- it looks to me like your model is doing what it should.

For reference, here's the plot from the question flipped around:

Flipped plot from question of y versus predicted y

There's some hint that it's slightly biased toward 0 (which as mentioned, may not be a problem), and also perhaps a slight suggestion of a nonlinear relationship (which might potentially be a problem).

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ It would help if we could get the figure presented better, but I wonder if this is the opposite of regression to the mean. The data are spreading over a wider range than the predictions. $\endgroup$ – gung - Reinstate Monica♦ Jul 28 '16 at 12:25
  • 1
    $\begingroup$ @gung That's how regression to the mean always works. The predictions are less spread than the data $\endgroup$ – Glen_b Jul 28 '16 at 12:57
  • $\begingroup$ @gung I have added a version of the plot from the question with axes interchanged. (I can't change the ratio of height and width without scraping data from the plot though -- not something to do today). $\endgroup$ – Glen_b Jul 28 '16 at 13:23
  • $\begingroup$ There is a confusion here (which may be mine). The way I think about regression to the mean is illustrated in my answer to Algorithms for automatic model selection: When you select based on a value, future values will be closer to the mean than what you picked. The data in the graph are further from the mean, AFAICT. Alternatively, the fitted slope is closer to 0 than the appropriate slope in the test set. This makes sense as RttM if slopes were selected by being closer to 0, but I gather that isn't what the OP did. $\endgroup$ – gung - Reinstate Monica♦ Jul 28 '16 at 21:47
  • $\begingroup$ @gung Regression to the mean occurs because $E(Y|X)-E(Y)$ is on average smaller in magnitude than $Y-E(Y)$ (e.g. mean-square sense). Say we have $E(Y|X)$ linear in $X$, so it is the linear regression line. See Senn's figures with Galton's data in figure 1 here (the dashed red lines are the regression lines, solid red are LOWESS curves). There (as is the case in Galton's own work) regression to the mean is illustrated by the fact that the regression line is closer to the mean (smaller slope) ...ctd $\endgroup$ – Glen_b Jul 28 '16 at 22:27
5
$\begingroup$

What these results tell you is something that is almost inevitable in this type of modeling: a model based on a training set will not fit a test set as well. The type of problem you face, with a difference in the slope of the relation between your linear predictor and the outcome variable between the training and test sets, seems to be one of calibration, as noted here in the context of logistic regression.

Your suggestion to "pivot" the slope of the line is similar to the general idea of "shrinking" regression coefficients to improve predictive ability on new data, but you would be better off using established methods like those provided by the rms package in R. Note that these efforts necessarily entail making a bias-variance tradeoff in predictive modeling. If you are unfamilar with that tradeoff, you should read An Introduction to Statistical Learning or a similar general reference.

Also, separate training and test sets might not be the most efficient way to use your data; developing the model on the entire data set and checking and adjusting calibration by bootstrap resampling may be better. Consider consulting Frank Harrell's course notes or his book for more detail.

Please check that your axes are labeled correctly. Typically one expects predicted values to be over-optimistic, with a wider range of predicted values than of observed. Also, the range of your results suggests that you might be using proportions as your outcome variable. If so, then there might be an issue if you used standard linear regression to develop your model.

| cite | improve this answer | |
$\endgroup$
3
$\begingroup$

I think most of the relevant information here has been provided by @EdM. Let me add a few thoughts:

Regarding your plot, I would put the predicted values on the x-axis and the observed values on the y-axis. That is the way scatterplots are more typically constructed and may help with interpretation. In addition, I would make the plot square and force the plotting area to range over the same possible values (say, $.1$ to $.9$) on both dimensions. Lastly, I would plot a LOWESS fit as well as the one to one line. These should make what is going on easier to see.

My guess is that your fitted model has too shallow a slope relative to the test data, but that the mean prediction is not biased (much). That shouldn't happen often. If your data were randomly split into train and test, they should be very similar, so the slopes wouldn't differ by much. This is also true given that you seem to have a lot of data. That should make estimates fairly stable. For me, these facts raise some questions:

  • Was the split random, or was it by some pre-existing group structure (e.g., males vs. females)?
  • Did you do a lot of fitting to get your model (such that it was overfitted)? More specifically, what was your modeling process?
  • Do you have a lot of variables or complex functions of variables in the original model (and/or relative to the amount of data)?

More information is needed here.

It is also possible that you are missing some curvature in the data.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ my modelling approach: Take monotonic transforms of all the variables, then used a forward subset algorithm to get the best k factor model for 1,...,k=# of predictors, then I calculated the Test MSE for each of the k models, and I picked the one with the lowest test MSE $\endgroup$ – dimebucker91 Jul 27 '16 at 23:51
  • 2
    $\begingroup$ @dimebucker91, that should lead to overfitting. $\endgroup$ – gung - Reinstate Monica♦ Jul 27 '16 at 23:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.