6
$\begingroup$

I am fitting a simple linear regression model with 4 predictors:

lm(Outcome ~ Predictor1 + Predictor2 + Predictor3 + Predictor4, data=dat.s)

I'm finding that the model predictions are consistently off as shown in this graph: scatterplot of predictions and residuals

The model clearly overestimates the low values and underestimates the high values, but the miss-estimation is very linear -- it seems like the model should be able to just adjust the slope and fit the data better. Why is that not happening? In case it helps, here are scatterplots of the the Outcome against each of the four Predictors: enter image description here

Using the car package outlierTest function did not identify any outliers.

$\endgroup$

2 Answers 2

12
$\begingroup$

You shouldn't plot the residuals against the predicator values because they are correlated, instead we plot against the fitted values, i.e. $\hat Y$. To see this, consider this simple data set:

x = runif(20)
y = rnorm(20)

Clearly $x$ and $y$ are unrelated. Now, we fitted a simple linear regression model

m = lm(y ~ x)

and plot the residuals against outcome and fitted values

plot(y, residuals(m))
plot(fitted.values(m), residuals(m))

to get:

enter image description here

Notice the plotting against the outcome $y$, doesn't give any insight. Hence, it's not really clear if there is something wrong with your model since we don't expect random scatter.

$\endgroup$
2
  • $\begingroup$ Nice example but I think this is a little more complicated because it involves four predictors and not just one. $\endgroup$ Sep 25, 2012 at 22:01
  • $\begingroup$ Thanks, the example was very helpful! I plotted the residuals against the predicted values and they are nicely evenly distributed. @MichaelChernick is right that my situation is a little more complicated and some of the predictors are correlated, but I think this was the crux of it. $\endgroup$
    – Dan M.
    Sep 25, 2012 at 23:34
5
$\begingroup$

What you are seeing is called "Regression towards the Mean" and is completely expected. Any time that there is variability in the data (and yours looks like it has a bunch) then the prediction values will on average be between the overall mean and the observed values. The plot you created, of the outcome vs the predicted values is not commonly done, for the reasons that you are seeing, it tends to confuse more than enlighten. It is more common to plot the predicted values against the residuals as this plot will show more randomness if the model is reasonable.

Edit to address comment below

csgillespie's example has been criticized for only including 1 predictor when the original question included 4. Here is some quick R code that can be run to show the same patterns with 4 predictors:

# simulated data, no relationship
df1 <- data.frame(y=rnorm(100), x1=rnorm(100), x2=rnorm(100),
        x3=rnorm(100), x4=rnorm(100))

fit1 <- lm( y ~ ., data=df1 )
#plot(df1$y, fitted(fit1), asp=1)
    scatter.smooth(df1$y, fitted(fit1), asp=1)
abline(0,1)
abline(h=mean(fitted(fit1)), col='lightgrey')
plot(df1$y, resid(fit1))
abline(h=0)

plot(fitted(fit1), resid(fit1))
abline(h=0)

# simulated data, relationship
library(MASS)
df2 <- as.data.frame( mvrnorm(100, mu=1:5, Sigma= matrix(.7,5,5)+diag(rep(.3,5))))
names(df2) <- c('y','x1','x2','x3','x4')

fit2 <- lm( y ~ ., data=df2 )
#plot(df2$y, fitted(fit2), asp=1)
    scatter.smooth(df2$y, fitted(fit2), asp=1)
abline(0,1)
abline(h=mean(fitted(fit2)), col='lightgrey')
plot(df2$y, resid(fit2))
abline(h=0)

plot(fitted(fit2), resid(fit2))
abline(h=0)


# real data
fit3 <- lm( Murder~Population+Income+Illiteracy+Frost, data=as.data.frame(state.x77))

scatter.smooth( state.x77[,'Murder'], fitted(fit3), asp=1)
abline(0,1)
abline(h=mean(fitted(fit3)), col='lightgrey')
plot(state.x77[,'Murder'], resid(fit3))
abline(h=0)

plot(fitted(fit3), resid(fit3))
abline(h=0)

Notice that the plots look very similar to those in the original question.

Notice also that in the plot of the original outcome vs. the fitted values that the points (and more so their trend) tend to fall between the $y=x$ line and the mean line. This is the idea of regression towards the mean as originally described by Galton See Here. The points are not randomly scattered about the $y=x$ line like the original poster assumed and what would happen without the regression towards the mean, rather they follow a linear trend along a line that represents proportianality between the $y=x$ line and the overall mean line, just as predicted by Galton. The term regression towards the mean (and variants) is sometimes used for other concepts (some closer related to the original than others) as can be seen in the above article, and that may be where some of the confusion comes from.

$\endgroup$
4
  • $\begingroup$ I think you are misinterpreting the plots. i think the one on the left is outcome verses prediction and the second is outcome verses residuals. So the OP is right that the residuals look peculiar because they do not scatter randomly across values of the outcome but rather have a linear trend. Regression toward the mean is seen easily in simple linear regression when you plot the line on covariate vs outcome axes. But that is not what is shown in either graph and there are 4 predictor covariates involved. $\endgroup$ Sep 25, 2012 at 20:33
  • $\begingroup$ @MichaelChernick, what you say is how I interpreted the plots. It is much more common to plot predicted vs. residuals (which will be independent) than outcome vs. residuals (which will be correlated as seen). My interpretation can be seen in the accepted answer above (but the accepted answer went the extra step of showing a demonstration). $\endgroup$
    – Greg Snow
    Sep 26, 2012 at 15:33
  • $\begingroup$ Yes that is fine. I think it is a simplified way of getting an important point across about what could be deceptive in a plot as csgillispie demonstrated. But I am not satisfied with the regression toward the mean explanation because the plot is a projection onto a plane of residuals verses outcome and not predicted values versus outcome and the residuals depend on 4 predictors rather than 1. $\endgroup$ Sep 26, 2012 at 15:59
  • $\begingroup$ @MichaelChernick, see the above edit to my response in answer to your comments. I think it is possibe that you are thinking of regression towards the mean in a different context than I am, so I added more information and a link. $\endgroup$
    – Greg Snow
    Sep 26, 2012 at 17:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.