I always thought the definition of residuals is the difference between the statistic and the observations. And, the definition of the prediction error is the difference between the true value and the prediction.

So, I was surprised when I saw on this wikipedia page it says

"...while lower-case y in the definition of the residuals..." under the section called "Residuals" and the sub-section called "Example". It seems to be showing the error because the lower case "y" is the true value, but it calls them the "residuals". Is that incorrect to call them "residuals"?


I find your post quite confusing, especially the part about the statistic and the example; how are they relevant here? Instead, let me provide my own understanding of [model] residuals and prediction errors.

A stochastic model includes an error term to allow the relationship between the variables to be stochastic (have some randomness to it) rather than deterministic (fixed, perfect). For example,

$$ y = \beta_0 + \beta_1 x + \varepsilon $$

implies a linear relationship between $y$ and $x$, up to some error $\varepsilon$. When the model is estimated, one gets the realized values of the model errors which are called [model] residuals (denoted $\hat\varepsilon$ or $e$):

$$ y = \hat\beta_0 + \hat\beta_1 x + \hat\varepsilon. $$

Now consider another expression which defines fitted values,

$$ \hat y := \hat\beta_0 + \hat\beta_1 x. $$

Together the above two expressions yield another expression for the [model] residuals; they are the difference between the actual and the fitted values of the dependent variable:

$$ \hat\varepsilon = y - \hat y. $$

Meanwhile, prediction errors arise in the context of forecasting. A prediction error is the difference between the realized value and the predicted value:

$$ e^{fcst} := y - y^{fcst}. $$

(Since the prediction $y^{fcst}$ is produced without (or before) having observed the realized value $y$, the prediction errors are generally not zero.)

Now to respond to your Wikipedia quote, let us look at it more closely:

<...> the capital letter Y is used in specifying the model, while lower-case y in the definition of the residuals; that is because the former are hypothesized random variables and the latter are actual data.

It only says that Y are hypothesized random variables and y are actual data. (If I had used this notation, I should have had capital Y in my first equation but lower-case y in my second equation and elsewhere.) The cited definition of residuals is five lines above the quoted text; there indeed is a formula including lower-case y and defining [model] residuals. If I interpret you correctly, you seem to have understood that y is called the residuals -- which it is not, if you read the Wikipedia quote carefully.

  • $\begingroup$ Nice work Mr. Hardy. I would add that in the context of time series analysis/data the residuals are in effect "one-period out prediction errors" based upon the parameters/model. If the parameters are concurrently being estimated then they are based upon the entire data set used for estimation. $\endgroup$ – IrishStat Jan 30 '16 at 18:54
  • $\begingroup$ @IrishStat, this is an interesting coincidence. When looking from the perspective of model fitting, the object is model residual; when looking from the perspective of forecasting, apparently the same object is prediction error. The difference is that in the first case the model is being estimated and at the same time it produces the residuals, while in the second case the model should be treated as given and the prediction errors are produced for a given model. Also, an important difference is what you already mentioned: estimation uses the entire sample while forecasting uses less than that. $\endgroup$ – Richard Hardy Jan 30 '16 at 19:01
  • $\begingroup$ Yeah, I was going to say that the "prediction errors" in time series seem like a special case of residuals. $\endgroup$ – Hunle Jan 30 '16 at 19:09
  • 4
    $\begingroup$ @Hunle, not exactly -- because prediction errors are produced by a fixed model (that has been fitted before making the forecasts) while residuals are produced at the same time as the model is being estimated. I think this is an important distinction. $\endgroup$ – Richard Hardy Jan 30 '16 at 19:19
  • $\begingroup$ @RichardHardy, Two things are not clear to me about the residual and the polulation error $\epsilon_i$ and I would like to learn those. 1. How can be we sure that $e_i$ estimates $\epsilon_i$, where an unknown random error component cannot be determined? 2. Generally,from the dataset we find out $$\widehat {Y{_i}}=\widehat{\beta_0}+\widehat{\beta_1}.X$$ and $$e_i=Y_i - \widehat {Y_i}$$, and nowhere we use $\epsilon$ in the analysis so how does the concept of $\epsilon$ arise in the population regression model? $\endgroup$ – vbm Jun 26 '18 at 1:15

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.