1
$\begingroup$

I'm trying to get a deeper understanding of how OLS works. One thing that I thought I understood is the difference between standard errors and residuals.

Here are two definitions

  1. Standard errors: The average distance that the observed values fall from the regression line.
  2. Residuals: The difference between the actual value and the value predicted by the model ($y_i - \hat y_i$) for any given point.

Where I always assumed that number 2 was unobservable (Actually in this post they claim it's the other way around: https://stats.stackexchange.com/a/232588/334202).

But if I run a simple regression in R like this I get both! So how can I think about this?

library(tidyverse)
library(broom)

mtsmall <- mtcars |>
  rownames_to_column(var="carnames") |> 
  as_tibble() |> 
  select(mpg,hp,wt)

model1 <- lm(mpg ~ hp, mtsmall)

mtsmall_predicted <- augment_columns(model1, mtsmall) |> 
  rename(.mpg_hat = .fitted) 

mtsmall_predicted |> head(5)

Output:

    mpg    hp    wt .mpg_…¹ .se.fit .resid   .hat .sigma .cooksd
  <dbl> <dbl> <dbl>   <dbl>   <dbl>  <dbl>  <dbl>  <dbl>   <dbl>
1  21     110  2.62    22.6   0.777 -1.59  0.0405   3.92 3.74e-3
2  21     110  2.88    22.6   0.777 -1.59  0.0405   3.92 3.74e-3
3  22.8    93  2.32    23.8   0.873 -0.954 0.0510   3.92 1.73e-3
4  21.4   110  3.22    22.6   0.777 -1.19  0.0405   3.92 2.10e-3
5  18.7   175  3.44    18.2   0.741  0.541 0.0368   3.93 3.89e-4
$\endgroup$
4
  • 1
    $\begingroup$ "Actually in this post they claim it's the other way around" That other post has it right. Where did you get your definition for 'residual'? $\endgroup$ Feb 2, 2023 at 8:37
  • $\begingroup$ From Stock &v Watson 5th ed. p. 149 (I just added the "i" subscript that they use in the book, but I don't think it makes a difference?). $\endgroup$
    – Tomas R
    Feb 2, 2023 at 8:42
  • 1
    $\begingroup$ When they say "the actual value" it means "the actual observed value" $\endgroup$ Feb 2, 2023 at 8:44
  • $\begingroup$ "the value predicted by the model" that is not a prediction for $y$ but an estimate of $E[y]$, the mean/expectation of the population of $y$'s. Residuals are the difference between the actual observed $y$ and the estimated $\hat{E(y)}$. Errors are the difference between the actual observed $y$ and the true population value $E(y)$ (that true value is not observed). $\endgroup$ Feb 2, 2023 at 8:47

1 Answer 1

1
$\begingroup$

You have errors and residuals, which are different and probably what you understood.

You seem to be speaking about 'standard error'. That is an indirect quantification of the 'error'. The standard error quantifies it in terms of an estimate for the standard deviation of the (estimated) sample distribution of that value.

$\endgroup$
3
  • $\begingroup$ So let me see if I got this right. Residual is the difference between the predicted value and the observed value (calculated from the data). Then the error is the difference from the mean in the population (most likely unobserved). Finally, the std. error is the estimation of that unobserved error? If this is true, why is not the residual then equal to the standard error? Many thanks! $\endgroup$
    – Tomas R
    Feb 2, 2023 at 8:47
  • 1
    $\begingroup$ @TomasR residuals and errors are varying. If you roll a 6 sided dice, then the difference between the actual dice roll and the mean 3.5, will be varying and 0.5, 1.5 or 2.5. The mean (root mean square more precisely) of those possible error values is the standard error. The standard error is estimated based on multiple residuals, that's why residuals may be different from the standard error. $\endgroup$ Feb 2, 2023 at 8:56
  • 1
    $\begingroup$ "std. error is the standard deviation of the sample distribution of the error" and in practice it is an estimation. $\endgroup$ Feb 2, 2023 at 8:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.