When running a multiple regression model in R, one of the outputs is a residual standard error of 0.0589 on 95,161 degrees of freedom. I know that the 95,161 degrees of freedom is given by the difference between the number of observations in my sample and the number of variables in my model. What is the residual standard error?

  • 2
    This question and its answers might help: Why do we say residual standard error? – Antoine Vernet Jul 27 '16 at 6:20
  • A quick question: Is "residual standard error" the same as "residual standard deviation"? Gelman and Hill (p.41, 2007) seem to use them interchangeably. – JetLag Jun 9 at 12:04
up vote 19 down vote accepted

A fitted regression model uses the parameters to generate point estimate predictions which are the means of observed responses if you were to replicate the study with the same $X$ values an infinite number of times (and when the linear model is true). The difference between these predicted values and the ones used to fit the model are called "residuals" which, when replicating the data collection process, have properties of random variables with 0 means.

The observed residuals are then used to subsequently estimate the variability in these values and to estimate the sampling distribution of the parameters. When the residual standard error is exactly 0 then the model fits the data perfectly (likely due to overfitting). If the residual standard error can not be shown to be significantly different from the variability in the unconditional response, then there is little evidence to suggest the linear model has any predictive ability.

Say we have the following ANOVA table (adapted from R's example(aov) command):

          Df Sum Sq Mean Sq F value Pr(>F)
Model      1   37.0   37.00   0.483  0.525
Residuals  4  306.3   76.57               

If you divide the sum of squares from any source of variation (model or residuals) by its respective degrees of freedom, you get the mean square. Particularly for the residuals:

$$ \frac{306.3}{4} = 76.575 \approx 76.57 $$

So 76.57 is the mean square of the residuals, i.e., the amount of residual (after applying the model) variation on your response variable.

The residual standard error you've asked about is nothing more than the positive square root of the mean square error. In my example, the residual standard error would be equal to $\sqrt{76.57}$, or approximately 8.75. R would output this information as "8.75 on 4 degrees of freedom".

  • 1
    I up-voted the answer from @AdamO because as a person who uses regression directly most often, that answer was the most straightforward for me. However, I appreciate this answer as it illustrates the notational/conceptual/methodological relationship between ANOVA and linear regression. – svannoy Mar 27 '16 at 18:40

Typically you will have a regression model looks like this: $$ Y = \beta_{0} + \beta_{1}X + \epsilon $$ where $ \epsilon $ is an error term independent of $ X $.

If $ \beta_{0} $ and $ \beta_{1} $ are known, we still cannot perfectly predict Y using X due to $ \epsilon $. Therefore, we use RSE as a judgement value of the Standard Deviation of $ \epsilon $.

RSE is explained pretty much clearly in "Introduction to Statistical Learning".

  • 1
    This should be the accepted answer. RSE is s just an estimate of the Standard Deviation of $\epsilon$, i.e. the residual. It's also known as the residual standard deviation (RSD), and it can be defined as $RSE = \sqrt{\frac{RSS}{(n-2)}}$ (e.g. see ISL page 66). – Amelio Vazquez-Reina Jul 23 '17 at 21:08

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.