# Does it make sense to study plots of residuals with respect to the dependent variable?

I would like to know whether it makes sense to study the plots of residuals with respect to the dependent variable when I've got a univariate regression. If it makes sense, what does a strong, linear, growing correlation between residuals (on the y-axis) and the estimated values of the dependent variable (on the x-axis) mean?

-
I'm not sure what you mean by "strong, linear, growing correlation". Can you show the plot? It's perfectly reasonable to plot residuals against fitted values. In general, you want there to be no relationship--a flat horizontal line running through the center. In addition, you want the vertical dispersion of the residuals to be constant from the left side of your plot to the right. –  gung Nov 19 '11 at 4:42
Hi. Thank you for your answer. This is the plot: img100.imageshack.us/img100/7414/bwages.png –  Luigi Nov 19 '11 at 11:26
That is perplexing. Let me make sure I understand: You ran a regression model, then plotted the residuals vs. the fitted values, and that's what you got, is that right? It shouldn't look like that. Can you edit your question and paste in the code you used for the model and the plot? –  gung Nov 19 '11 at 18:56
You understood right. I'm sorry, but I don't know how to retrieve the code, I ran the regression and plotted the residuals with the program Gretl. –  Luigi Nov 19 '11 at 19:33
I didn't initially see the comment by @mark999 when I wrote my answer below. I think his suspicion is correct, that this is is residuals vs. y-values. Luigi, redo your graph - don't try to interpret it when you may be wrong about what the variables are. –  Michael Bishop Nov 21 '11 at 3:14

Suppose that you have the regression $y_i = \beta_0 + \beta_1 x_i + \epsilon_i$, where $\beta_1 \approx 0$. Then, $y_i - \beta_0 \approx \epsilon_i$. The higher the $y$ value, the bigger the residual. On the contrary, a plot of the residuals against $x$ should show no systematic relationship. Also, the predicted value $\hat{y}_i$ should be approximately $\hat{\beta}_0$---the same for every observation. If all the predicted values are roughly the same, they should be uncorrelated with the errors.

What the plot is telling me is that $x$ and $y$ are essentially unrelated (of course, there are better ways to show this). Let us know if your coefficient $\hat{\beta}_1$ is not close to 0.

As better diagnostics, use a plot of the residuals against the predicted wage or against the $x$ value. You should not observe a distinguishable pattern in these plots.

If you want a little R demonstration, here you go:

y      <- rnorm(100, 0, 5)
x      <- rnorm(100, 0, 2)
res    <- lm(y ~ x)$residuals fitted <- lm(y ~ x)$fitted.values
plot(y, res)
plot(x, res)
plot(fitted, res)

-
+1 I guess you beat me to it. –  Adam Nov 21 '11 at 2:29
This does not mean that only because of $\beta_1=0$, it might be possible that the model needs more explanatory variables, such as polynomial terms. –  Biostat Nov 21 '11 at 10:31

Assuming the estimated model is correctly specified...

Let's denote $P_X=X(X'X)^{-1}X'$, the matrix $P_X$ is a projection matrix, so $P_X^2=P_X$ and $P_X'=P_X$.

$Cov(\hat{Y},\hat{e})=Cov(P_XY,(I-P_X)Y)=P_XCov(Y,Y)(I-P_X)'=\sigma^2P_X(I-P_X)=0$.

So the scatter-plot of residuals against predicted dependent variable should show no correlation.

But!

$Cov(Y,\hat{e})=Cov(Y,(I-P_X)Y)=Cov(Y,Y)(I-P_X)'=\sigma^2(I-P_X)$.

The matrix $\sigma^2(I-P_X)$ is a projection matrix, its eigenvalues are 0 or +1, it's positive semidefinite. So it should have non-negative values on the diagonal. So the scatter-plot of residuals against original dependent variable should show positive correlation.

As far as i know Gretl produces by default the graph of residuals against original dependent variable (not the predicted one!).

-
I appreciate the different possibility. This is where some knowledge of Gretl is helpful. I wonder however, how plausible it is that this is as the real answer. Using my simulated data, I correlated and plotted residuals vs. original dv; r=.22 and the plot looks a lot like my 3rd plot, not the question plot. Of course, I worked up those data to check the plausibility of my story--they may not be appropriate to check yours. –  gung Nov 21 '11 at 18:32
@gung what do you mean you used your simulated data? –  Michael Bishop Nov 21 '11 at 19:35
@MichaelBishop if you look at my answer, you see that I simulated data to try out my story to see if it would look like the posted plot. My code and plots are presented. Since I specified the seed, it is reproducible by anyone with access to R. –  gung Nov 21 '11 at 21:58

Is it possible you are confusing fitted/predicted values with the actual values?

As @gung and @biostat have said, you hope there is no relationship between fitted values and residuals. On the other hand, finding a linear relationship between the actual values of the dependent/outcome variable and the residuals is to be expected and is not particularly informative.

Added to clarify the previous sentence: Not just any linear relationship between residuals and actual values of the out come is to be expected... For low measured values of Y, the predicted values of Y from a useful model will tend to be higher than the actual measured values, and vice versa.

-
The implication of what you're saying is that, if values are consistently underpredicted at low values of Y, and consistently overpredicted at high values of Y, that's OK. That's a problem, right? –  rolando2 Nov 20 '11 at 19:54
@rolando2, I have not implied what you say I've implied though perhaps I should clarify my answer. As you said, consistently underpredicting at low vales of Y and overpredicting at high values of Y would be a sign of a very bad model. I imagined the opposite, overpredicting at low values of Y and underpredicting at high values of Y. This phenomena is common, and is to be expected roughly in proportion to how much of the variance in the dependent variable you are able to explain. Imagine you lack any variables which predict Y, so you always use the mean as your prediction –  Michael Bishop Nov 21 '11 at 0:02
what you've said makes sense to me, except for one thing. I'm having trouble imagining that a trend as strong as the one Luigi has shown would ever show up in a sound or desirable solution, even if the trend went from upper left to lower right. –  rolando2 Nov 21 '11 at 1:21
@rolando2, Residuals are typically defined as observed - fitted, therefore negative residuals are over-predictions. In a properly specified model with little explanatory power - I'm a social scientist so I see these all the time - there will be a strong positive relationship between residuals and the observed outcome values. If this is a residuals vs. actual plot, then a trend from upper left to lower right, would be the signal of a badly mis-specified model which you initially worried about. –  Michael Bishop Nov 21 '11 at 3:07
Thanks for that explanation. –  rolando2 Nov 21 '11 at 3:17

The answers offered are giving me some ideas about what's going on here. I do believe there may have been some mistakes made by accident. See if the following story makes sense: To start, I think there is probably a strong relationship between X & Y in the data (here's some code and a plot):

set.seed(5)
wage <- rlnorm(1000, meanlog=2.3, sdlog=.5)
something_else <- .7*wage + rnorm(1000, mean=0, sd=1)
plot(wage, something_else, pch=3, col="red", main="Plot X vs. Y")


But by mistake Y was predicted just from the mean. Compounding this, the residuals from the mean only model are plotted against X, even though what was intended was to plot against the fitted values (code & plot):

meanModel <- lm(something_else~1)
windows()
plot(wage, meanModel$residuals, pch=3, col="red", main="Plot of residuals from Mean only Model against X") abline(h=0, lty="dotted")  We can fix this by fitting the appropriate model and plotting the residuals from that (code & plot): appropriateModel <- lm(something_else~wage) windows() plot(appropriateModel$fitted.values, appropriateModel$residuals, pch=3, col="red", main="Plot of residuals from the appropriate\nmodel against fitted values") lines(lowess(appropriateModel$residuals~appropriateModel\$fitted.values))


This seems like just the kinds of goof-ups I made when I was starting.

-

This graph indicates that the model you fitted is not good. As @gung said in the first comments on the main question that there should be no relationship between predicated response and residual.

" an analyst should expect a regression model to err in predicting a response in a random fashion; the model should predict values higher than actual and lower than actual with equal probability. See this"

I would recommend first plot response vs independent variable to see the relationship between them. It might be reasonable to add polynomial terms in the model.

-

Isn't this what happens if there is no relationship between the X & Y variable? From looking at this graph, it appears you are essentially predicting Y with it's mean.

-