24
$\begingroup$

While these two ubiquitous terms are often used synonymously, there sometimes seems to be a distinction. Is there indeed a difference, or are they exactly synonymous?

$\endgroup$
22
$\begingroup$

Errors pertain to the true data generating process (DGP), whereas residuals are what is left over after having estimated your model. In truth, assumptions like normality, homoscedasticity, and independence apply to the errors of the DGP, not your model's residuals. (For example, having fit $p+1$ parameters in your model, only $N-(p+1)$ residuals can be independent.) However, we only have access to the residuals, so that's what we work with.

$\endgroup$
  • 9
    $\begingroup$ (+1) Residuals can be considered estimates of the errors. $\endgroup$ – Scortchi - Reinstate Monica Jan 14 '15 at 17:09
  • $\begingroup$ @ABC, DGP stands for data generating process. Even if your model is appropriate & reflects the true structure of the DGP, the residuals won't necessarily be normal, homoscedastic & independent if the underlying errors weren't. $\endgroup$ – gung - Reinstate Monica Apr 28 '15 at 14:01
  • $\begingroup$ @Scortchi Hi, do you happen to have any references that expand on your comment? I'm trying to understand why exactly residuals can be used as estimates of the error, because I see people checking normality of residuals in regression analysis when the actual assumption is normality of error, and I don't know precisely why that is valid. $\endgroup$ – Austin Aug 31 '17 at 14:30
  • $\begingroup$ @Austin, if you are still interested in that, you should ask a new question. $\endgroup$ – gung - Reinstate Monica Aug 12 '19 at 16:51
16
$\begingroup$

An error is the difference between the observed value and the true value (very often unobserved, generated by the DGP).

A residual is the difference between the observed value and the predicted value (by the model).

$\endgroup$
  • $\begingroup$ In response to the error being "very often unobserved", in reality we could never observe the absolute true value. Every measurement device has finite error inherent in the measurement device. The best you could claim is that the true value at a point equals whatever measured plus/minus the observational error, or measurement error, e.g., my ruler can increment distances down to 1 mm, so the best I can measure the true length of my pencil is within $\pm$0.1mm. Note this differs from any systematic error, e.g., my ruler was somehow stretched out during manufacturing and measures too long. $\endgroup$ – Steven C. Howell Nov 21 '17 at 14:22
  • $\begingroup$ This link (ece.rochester.edu/courses/ECE111/error_uncertainty.pdf) provides a good explanation with references to Bevington's and Taylor's texts on the subject. $\endgroup$ – Steven C. Howell Nov 21 '17 at 14:26
  • $\begingroup$ In machine learning language, is residue the training error and error is test error? $\endgroup$ – Charles Chow Dec 6 '18 at 21:06
  • $\begingroup$ @CharlesChow It depends on which dataset you use. If you use training set, then it is training error; if you use test set, then it is test error. $\endgroup$ – Leopold W. Nov 15 '19 at 6:13
7
$\begingroup$

Error term is a theoretical concept that can never be observed, but the residual is a real world value that is calculated for each time a regression is done

$\endgroup$
0
$\begingroup$

Error of the data set is the differences between the observed values and the true / unobserved values. Residual is calculated after running the regression model and is the differences between the observed values and the estimated values.

$\endgroup$
  • 1
    $\begingroup$ This doesn't need a downvote, it isn't incorrect. I think the reason you got a downvote is because your answer is the same as several answers that already exist on this thread & doesn't add any new information. That will often be the case, when it is, you just don't post your answer on that thread. Find another thread where you can make a contribution & answer there. $\endgroup$ – gung - Reinstate Monica Jan 1 at 14:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.