26
$\begingroup$

While these two ubiquitous terms are often used synonymously, there sometimes seems to be a distinction. Is there indeed a difference, or are they exactly synonymous?

$\endgroup$
26
$\begingroup$

Errors pertain to the true data generating process (DGP), whereas residuals are what is left over after having estimated your model. In truth, assumptions like normality, homoscedasticity, and independence apply to the errors of the DGP, not your model's residuals. (For example, having fit $p+1$ parameters in your model, only $N-(p+1)$ residuals can be independent.) However, we only have access to the residuals, so that's what we work with.

| cite | improve this answer | |
$\endgroup$
  • 15
    $\begingroup$ (+1) Residuals can be considered estimates of the errors. $\endgroup$ – Scortchi - Reinstate Monica Jan 14 '15 at 17:09
  • $\begingroup$ @ABC, DGP stands for data generating process. Even if your model is appropriate & reflects the true structure of the DGP, the residuals won't necessarily be normal, homoscedastic & independent if the underlying errors weren't. $\endgroup$ – gung - Reinstate Monica Apr 28 '15 at 14:01
  • $\begingroup$ @Scortchi Hi, do you happen to have any references that expand on your comment? I'm trying to understand why exactly residuals can be used as estimates of the error, because I see people checking normality of residuals in regression analysis when the actual assumption is normality of error, and I don't know precisely why that is valid. $\endgroup$ – Austin Aug 31 '17 at 14:30
  • $\begingroup$ @Austin, if you are still interested in that, you should ask a new question. $\endgroup$ – gung - Reinstate Monica Aug 12 '19 at 16:51
  • $\begingroup$ What is p and N in the answer please elaborate it @gung-ReinstateMonica $\endgroup$ – saravanan saminathan Mar 24 at 11:28
20
$\begingroup$

An error is the difference between the observed value and the true value (very often unobserved, generated by the DGP).

A residual is the difference between the observed value and the predicted value (by the model).

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ In response to the error being "very often unobserved", in reality we could never observe the absolute true value. Every measurement device has finite error inherent in the measurement device. The best you could claim is that the true value at a point equals whatever measured plus/minus the observational error, or measurement error, e.g., my ruler can increment distances down to 1 mm, so the best I can measure the true length of my pencil is within $\pm$0.1mm. Note this differs from any systematic error, e.g., my ruler was somehow stretched out during manufacturing and measures too long. $\endgroup$ – Steven C. Howell Nov 21 '17 at 14:22
  • $\begingroup$ This link (ece.rochester.edu/courses/ECE111/error_uncertainty.pdf) provides a good explanation with references to Bevington's and Taylor's texts on the subject. $\endgroup$ – Steven C. Howell Nov 21 '17 at 14:26
  • $\begingroup$ In machine learning language, is residue the training error and error is test error? $\endgroup$ – Charles Chow Dec 6 '18 at 21:06
  • $\begingroup$ @CharlesChow It depends on which dataset you use. If you use training set, then it is training error; if you use test set, then it is test error. $\endgroup$ – Leopold W. Nov 15 '19 at 6:13
7
$\begingroup$

Error term is a theoretical concept that can never be observed, but the residual is a real world value that is calculated for each time a regression is done

| cite | improve this answer | |
$\endgroup$
1
$\begingroup$

Error of the data set is the differences between the observed values and the true / unobserved values. Residual is calculated after running the regression model and is the differences between the observed values and the estimated values.

| cite | improve this answer | |
$\endgroup$
  • 3
    $\begingroup$ This doesn't need a downvote, it isn't incorrect. I think the reason you got a downvote is because your answer is the same as several answers that already exist on this thread & doesn't add any new information. That will often be the case, when it is, you just don't post your answer on that thread. Find another thread where you can make a contribution & answer there. $\endgroup$ – gung - Reinstate Monica Jan 1 at 14:00
1
$\begingroup$

Error term is an unknown value that could never be known unless the DGP is known. Therefore, theoretically, one can generate a variable x from say a Normal random variable and the error from a normal random variable. Then construct the variable $y$ as follow

$$ y_t=\beta x_t+e_t $$

Here, $e_t$ stands for error term the difference between the true variable $y_t$ and the expected value $\beta x_t$.

$\beta$ is unknown, once the beta is estimated then we get

$$ y_t=\hat{\beta} x_t+\hat{e}_t $$

Then $\hat{e}_t$ is not the error anymore it is the residual, the difference between the true value $y_t$ and the estimated value $\hat{\beta} x_t:=\hat{y}_t$.

This comes in line with another question, what is the difference between the Mean Squared Error and Mean Squared residual. There is nothing called MSR: Means squared residual.

$$ MSR=\frac{1}{n}\sum_{i=1}^{n}\hat{e}_i $$

However, many practitioners treat them the same. MSE is a theoretical concept that is always translated to MSR by practitioners due to their unfamiliarity between theory and practice.

$$ MSE=E(e_{t}^{2}) $$

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.