Had a question on a quiz I chose a (I and III) but the teacher said d.(all the above)

Which of the following statements about residuals are true?

I. The means of the residuals is always zero

II. The regression line for a residual plot is a horizontal line.

III. A definite pattern in the residual plot is an indication that a nonlinear model will show a better fit to the data than the LSRL

I don't think II is correct. If you have three residuals (-1,0,1) then you would have a line threw all three points.

Could someone help me understand what my mistake is? The regression line minimizes the distance between the points and the line, thus with my example (-1,0,1) that fits on a straight line with r =1.

  • $\begingroup$ did "d" mean "all of the above?" $\endgroup$ – Peter Ellis Jan 11 '13 at 4:29
  • 4
    $\begingroup$ Can you contrive a case where, after the ordinary least squares fit, you have three equally spaced residuals (-1,0,1) in that order? If you think through why this is impossible it might help. $\endgroup$ – Peter Ellis Jan 11 '13 at 4:32
  • $\begingroup$ @PeterEllis Do you know of a good online resources that will walk me threw the mathematics and not just state it without any proof? $\endgroup$ – MaoYiyi Jan 11 '13 at 4:37
  • 2
    $\begingroup$ "Residual plot" typically refers to a plot of the residuals on the y axis and an independent variable on the x-axis. If you then draw a regression line of the residual on the IV, it has to be flat because the linear relationship between the IV and the DV has already been accounted for in the original regression. $\endgroup$ – Peter Flom Jan 11 '13 at 11:54
  • 1
    $\begingroup$ Eight such residual plots appear in the middle of my post at stats.stackexchange.com/questions/46185/…. The figure is a scatterplot matrix and the relevant residual plots have regression lines drawn in red: all are horizontal, as @Peter Flom argues. (A ninth red line, at the lower right, would not be possible to obtain in ordinary circumstances, because it relies on information that is not usually available.) $\endgroup$ – whuber Jan 11 '13 at 15:29

Suppose your data is $\{(x_i,y_i)\}$ with non constant $x_i$, and you choose an arbitrary line $y=\beta_0 + \beta_1 x $ with residuals $r_i = y_i - \beta_0 - \beta_1 x_i$ associated with it.

Now suppose that the least squares regression line of $\{r_i\}$ on $\{x_i\}$ gives $r=\hat\beta_2 + \hat\beta_3 x $. Since this is a least squares regression line, if at least one of $\hat\beta_2$ or $\hat\beta_3$ is non-zero then you have $$\sum_i(r_i - \hat\beta_2 - \hat\beta_3 x_i)^2 \lt \sum_i r_i^2$$ or written another way $$\sum_i(y_i - (\beta_0 + \hat\beta_2) - (\beta_1 + \hat\beta_3)x_i)^2 \lt \sum_i(y_i - \beta_0 - \beta_1 x_i)^2.$$

So your original arbitrary line can only minimise the sum of squares of the residuals if both $\hat\beta_2$ and $\hat\beta_3$ are zero. $\hat\beta_3=0$ means a horizontal regression line of the residuals.

| cite | improve this answer | |
  • $\begingroup$ Which book would explain statistics like this? $\endgroup$ – MaoYiyi Jan 13 '13 at 9:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.