2
$\begingroup$

According to what I read online, the error term and the residual are often interchangeable. Please let me know if my understanding below is correct:

However, the difference is that the error term is the difference between our predicted value and the ACTUAL population value. So for instance, if we are to measure the relationship between salary level and experience level for a 35 year old male in the US, we wouldn't be able to get the data for ALL of the millions of 35 year old males in the US. However, we can get a sample population of that whole population. So therefore, our regression of Y would include an error term which indicates the value of what we have and the ACTUAL value of the population that we CANNOT OBSERVE. However, the RESIDUAL is the difference between the actual regression line and the actual POINTS of the scatter plot of data that we have that we have actually OBSERVED from going out and collecting data from some source we may have.

Is my understanding correct?

$\endgroup$
3
  • 3
    $\begingroup$ To see why the concept of observability is not at the root of this, suppose you actually did have a complete and accurate database of salary and experience for 35 y.o. US males. Because these two variables would not exhibit a linear (or even monotonic) relationship, there would still be residuals, there would still be error terms, and the two would not necessarily coincide because (1) the residuals depend on the procedure you use for fitting the model and (2) by taking the view that this dataset is the entire population of interest, the errors no longer are random variables. $\endgroup$
    – whuber
    May 17 '19 at 14:54
  • $\begingroup$ So then in this case, what exactly IS the error term and what is the residual? I've always understood residual is just the "physical distance" between the data points from the best fit line when doing an OLS regression. It reminds me of like high school math (square root of (x2-x1)^2+ (y2-y1)^2) $\endgroup$
    – John
    May 17 '19 at 15:21
  • 1
    $\begingroup$ Simply put, the error term is a construct in a model of the population or process and the residual is the difference between an observation and the value assigned to that observation by your regression procedure. As a concrete example, suppose a single value $X$ is randomly selected from a population known to have a Normal distribution with unknown mean $\mu;$ that is, the model is that $X=\mu+\epsilon.$ The error is $\epsilon=X-\mu.$ The OLS estimate of $\mu$ is $X$ itself, with residual $X-X=0.$ In this case the residual and the error almost surely are unequal. $\endgroup$
    – whuber
    May 17 '19 at 15:28
2
$\begingroup$

Your wording seems to imply that the error term exists because we deal with samples and the error term captures information about the non-sampled part of the population. That's not correct. The error term in a regression model represents factors other than the observed variables included in the model as $X$'s (explanatory/independent variables) that affect the dependent variable $Y$. Regression model (e.g., $y = \beta_{0} + \beta_{1}x + \epsilon$) begins from assuming what the relationship between $X$ and $Y$ variables is in the population, so the error term exists even in the population model. The model you end up estimating with sample data allows you to estimate the parameters of that population model.

So the error term is NOT the difference between observed and predicted values of $Y$. Repeating myself, it represents unobserved factors affecting $Y$.

Once the population regression model is assumed, we proceed to estimating the model with randomly sampled data. The estimation/fitting procedure we use estimates the values of $\beta$'s and we can then compute predicted/fitted values of $Y$ based on those estimated values of $\beta$'s and the observed values of $X$'s. The estimated regression equation takes on the form: $y_{i} = \hat\beta_{0} +\hat\beta_{1}x_{i} + \hat\epsilon_{i}$, with those hats denoting estimated values. $\hat\epsilon_{i}$'s are the residuals in the estimated equation (differences between observed and predicted values of $Y$ for each individual in the sample), while $\epsilon$'s are the errors in the equation containing population parameters $\beta_{0}$ and $\beta_{1}$. The errors are not observable, while the residuals are computed from the data.

$\endgroup$
2
  • $\begingroup$ I get what your point but sometimes when I go online and watch videos like those by Ben Lambert, he often times seems to use them interchangeably. He would give the equation: y=β0+β1x+ϵ (w/o the hat) and then call it the error term. However, I've also seen people say that "ϵ" is the residual so I'm confused. This website right here is where I got my information about this original post "statisticshowto.datasciencecentral.com/error-term". I simply want to know: if I gather "this" set of data with certain variables, run regression on it, is the "ϵ" I get the residual or error term? $\endgroup$
    – John
    May 17 '19 at 12:37
  • $\begingroup$ Yes, terms are often used interchangeably, but the technical definition is what is in the answer above. If you run a regression and compute a difference observed and predicted values of $Y$ for each observation, they are residuals. $\endgroup$
    – AlexK
    May 17 '19 at 19:35

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.