In section 3.2.3 of Elements of statistical learning, the authors wrote in equation 3.23

$$ Y = X\beta + \epsilon $$

but didn't give a name for $\epsilon$. Then it states that the "residuals" are $$ r_i = y_i - x_i \hat{\beta} $$

What is the formal terminology for $\epsilon$ and how does it relate to the residuals, $r_i$?

My understanding is that $\epsilon$ represents the deviation between $Y$ and $X\beta$. and $r_i$ represents the deviation between $Y$ and $X \hat{\beta} = \hat{Y}$. So is $r$ an approximation of $\epsilon$?

  • 2
    $\begingroup$ The notation $\epsilon$ (epsilon) is often tied to the name "error term" and alliteration is one element there, as epsilon was (and is) the Greek letter $e$. Although historical, the idea of an error term or disturbance (for which $u$ is also common notation) is broadly a nuisance insofar as it leads to an expectation among learners (all of us, some of the tme) that other quite different models can be thought of as including error terms, yet often they can't (or can't easily). $\endgroup$ – Nick Cox May 21 at 6:43

The two are closely related and I think you have the right idea. The error term is the difference between observed value and the theoretical expected value of the model at the point defined by the independent variables. The residuals of an observed value are the differences between the observed value and the estimated values of the model at the same point.

So in the case where the model parameters are unbiased and an approximation of the "real" underlying distribution, r will be an approximation of $\epsilon$, and furthermore the distribution of r's for a given vector of $X$ will an approximation to the real error distribution.

The wiki page gives a more thorough definition.


| cite | improve this answer | |
  • $\begingroup$ Hmm I see. Is the theoretical model also linear? $\endgroup$ – user5965026 May 21 at 5:34
  • 1
    $\begingroup$ @user5965026 the true ("theoretical") model can be anything, linear models are pften used as examples because of being most simple and most popular. $\endgroup$ – Tim May 21 at 6:54
  • $\begingroup$ @Tim Ah I see. But I was specifically wondering, when we write $Y = X\beta + \epsilon$, we are saying that $f(\beta) = X\beta$ is the true model? And if so, in this case we are saying the true model is linear? If so, I am confused how we can say the "true model" is linear, if the true model isn't observed. $\endgroup$ – user5965026 May 21 at 18:10
  • $\begingroup$ Sorry above should say $f(X)$ instead of $f(\beta)$ $\endgroup$ – user5965026 May 21 at 18:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.