Here is a question in the book of Freedman p.189. enter image description here The answers provided are (a) ( i)&(ii) (b) not used (c) something wrong

My biggest confusion is the answer for (c), what is wrong with this residual plot? The second biggest headache is (a) could be an answer represented by both (i) and (ii). Intuitive it seems ok. But I am unsure how it get calculated as 1000 from (i), (ii).

Hope to hear some helps. I wish an explanation at 1st year level without invoking the term endogeneity/exogenous etc.


Picture (iii) denotes a clear linear relationship, which should have been detected and exploited by the initial linear regression; that lead to these residuals. That is why something went wrong.

The absolute observed errors in (i) and (ii) lie between 0 and 2000. None comes close to 5000.


Let there be $n$ individuals. If the linear regressions follows the form of

$$y_{i} = \beta_0+\beta_1x_{i}+\epsilon_{i}$$ and $\epsilon_{i}$ follows a linear pattern as well, i.e. $\epsilon_{i}={\beta_\epsilon}_0 + {\beta_\epsilon}_1x_{i}+{\epsilon_2}_{i},$ then plugging this error term into the above linear regression yields $$y_{i} = \beta_0+\beta_1x_{i}+{\beta_\epsilon}_0 + {\beta_\epsilon}_1x_{i}+{\epsilon_2}_{i}.$$ You can easily rearrange the above expression into $$y_{i} = (\beta_0+{{\beta_\epsilon}_0})+(\beta_1+{\beta_\epsilon}_1)x_{i}+ {\epsilon_2}_{i}.$$ Note, this too is linear. The algorithm will find those $\beta$s which minimize the sum of squared errors, and that is the one with no linear part in $\epsilon$.

| cite | improve this answer | |
  • $\begingroup$ why residual plot should not present a linear relationship. On freedman 185 I saw a similar plot for the RMS error and it looks similar to the plot in c). I believe your answer is correct. But I do not clearly understand your explanation on why iii) is wrong. (I did insufficient studies on this topic and perhaps forgotten after studying regression analysis 10 years ago) Could you elaborate? $\endgroup$ – math101 Apr 25 '18 at 12:18
  • $\begingroup$ Let's assume, in the given pictures the x-axis is the predicted value, the y axis the residual. In (iii) a large prediction goes along with a large residual, a small prediction goes along with a negative prediction, both boosting the "least squares". If the modell predicted every prediction larger, then the residual would be less positive and if it predicted each prediction smaller, the residuals would be less negative. Does that make sense? I take for granted, that we talk about OLS regression (everything else would have to be announced) which optimizes for small "squares". $\endgroup$ – Bernhard Apr 25 '18 at 12:36
  • $\begingroup$ basically you mean the residuals go up with the prediction, which indicate some linear relationship which could leads to the issues: autocorrelated errors? By the way, I do not understand the sentence "If the modell predicted every prediction larger, then the residual would be less positive and if it predicted each prediction smaller, the residuals would be less negative."... $\endgroup$ – math101 Apr 25 '18 at 12:41
  • $\begingroup$ I have tried to make it more clear by my addition to the above answer. $\endgroup$ – Bernhard Apr 27 '18 at 12:56
  • 1
    $\begingroup$ It is lots clearer now and I’ve ticked. $\endgroup$ – math101 May 1 '18 at 0:52

Well, let's think about the broader picture first. What does the residual set graph tell us? It's graphing the relationship between some $x_{i}$, an independent variable used in the regression, and the error $\hat{e}_{i} = \hat{Y}_{i} - Y_{i}$. This graph provides a visual aid to tease out relationships between the unexplained component and the non-stochastic elements of the estimation process.

Okay, so how is RMSE calculated?

$RMSE = \sqrt[]{\frac{\sum_{n=1}^{N}(\hat{Y_{i}}-Y_{i})^{2}}{N}}$.

So, just looking at the set of values the residual can take, I see you have values of $\hat{e}_{i}$ which approach the absolute value of 2000. It's harder to decisively say without knowing the number of observations but the RMSE seemingly exceeds 1000 in either case. So, if I had to go with either a or b, I'd choose b.

The third case is a standard example of something we call endogeneity, i.e. $E[e_{i}|x_{i}] \neq 0$.

Edit: It looks like the image is the set of potential residuals realizations. If so, just plug in the largest set of values for $\hat{e}_{i}$ possible for a low $n$ regression and you will see the RMSE is larger than 1000.

| cite | improve this answer | |
  • $\begingroup$ I believe you are correct as well and this is an example of endogeneity. However I do not see how you could visualise $E[e_i|x_i]\neq 0$. Could you explain? This is exercise 2 p.189 of the book by Freedman. There is no other detail about RMSE given. The chapter label as RMS errors this is perhaps why it is labeled that way. $\endgroup$ – math101 Apr 25 '18 at 12:42
  • 1
    $\begingroup$ @JuliusBilly I understand the curves in the plots as reasonable borders of a scatter plot, not as the residuals liing on that lines. So the residuals lie anywhere in between -x and +x with x close but smaller 2000. It may well be, that many of those points reflect residuals of zero or close to zero. How can you tell "certainly" that the RSME exceeds 1000? $\endgroup$ – Bernhard Apr 25 '18 at 12:43
  • $\begingroup$ @math101 Pick a value of $x_{i}$ on the x-axis for the third image to the left or right of the set of values the residuals can take. Does its center lie about the x-axis? No. Hence, this is endogenous. If the set's center completely sat about the x-axis then you would have exogeneity. $\endgroup$ – JuliusBilly Apr 25 '18 at 12:49
  • $\begingroup$ @Bernhard your explanation seems more likely. I will edit my answer above. It does seem like the convex hull of residual values or something of that flavor. So, my reasoning goes as following (assuming you are correct and the number of observations we see is low enough): if that is the set of values the residual can take, then the residual can be something like 1500. If you plug this into the RMSE formula you get a value much larger than 1000. $\endgroup$ – JuliusBilly Apr 25 '18 at 12:52
  • $\begingroup$ @JuliusBilly I do not understand the statement " Does its center lie about the x-axis?" You picked a point $x_i$, what do you mean by this point center lie about x-axis? $\endgroup$ – math101 Apr 25 '18 at 13:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.