49
$\begingroup$

When running a multiple regression model in R, one of the outputs is a residual standard error of 0.0589 on 95,161 degrees of freedom. I know that the 95,161 degrees of freedom is given by the difference between the number of observations in my sample and the number of variables in my model. What is the residual standard error?

$\endgroup$
2
  • 3
    $\begingroup$ This question and its answers might help: Why do we say residual standard error? $\endgroup$ Jul 27, 2016 at 6:20
  • $\begingroup$ A quick question: Is "residual standard error" the same as "residual standard deviation"? Gelman and Hill (p.41, 2007) seem to use them interchangeably. $\endgroup$
    – JetLag
    Jun 9, 2018 at 12:04

5 Answers 5

30
$\begingroup$

A fitted regression model uses the parameters to generate point estimate predictions which are the means of observed responses if you were to replicate the study with the same $X$ values an infinite number of times (and when the linear model is true). The difference between these predicted values and the ones used to fit the model are called "residuals" which, when replicating the data collection process, have properties of random variables with 0 means.

The observed residuals are then used to subsequently estimate the variability in these values and to estimate the sampling distribution of the parameters. When the residual standard error is exactly 0 then the model fits the data perfectly (likely due to overfitting). If the residual standard error can not be shown to be significantly different from the variability in the unconditional response, then there is little evidence to suggest the linear model has any predictive ability.

$\endgroup$
1
35
$\begingroup$

Say we have the following ANOVA table (adapted from R's example(aov) command):

          Df Sum Sq Mean Sq F value Pr(>F)
Model      1   37.0   37.00   0.483  0.525
Residuals  4  306.3   76.57               

If you divide the sum of squares from any source of variation (model or residuals) by its respective degrees of freedom, you get the mean square. Particularly for the residuals:

$$ \frac{306.3}{4} = 76.575 \approx 76.57 $$

So 76.57 is the mean square of the residuals, i.e., the amount of residual (after applying the model) variation on your response variable.

The residual standard error you've asked about is nothing more than the positive square root of the mean square error. In my example, the residual standard error would be equal to $\sqrt{76.57}$, or approximately 8.75. R would output this information as "8.75 on 4 degrees of freedom".

$\endgroup$
1
  • 2
    $\begingroup$ I up-voted the answer from @AdamO because as a person who uses regression directly most often, that answer was the most straightforward for me. However, I appreciate this answer as it illustrates the notational/conceptual/methodological relationship between ANOVA and linear regression. $\endgroup$
    – svannoy
    Mar 27, 2016 at 18:40
29
$\begingroup$

Typically you will have a regression model looks like this: $$ Y = \beta_{0} + \beta_{1}X + \epsilon $$ where $ \epsilon $ is an error term independent of $ X $.

If $ \beta_{0} $ and $ \beta_{1} $ are known, we still cannot perfectly predict Y using X due to $ \epsilon $. Therefore, we use RSE as a judgement value of the Standard Deviation of $ \epsilon $.

RSE is explained pretty much clearly in "Introduction to Statistical Learning".

$\endgroup$
2
  • 7
    $\begingroup$ This should be the accepted answer. RSE is s just an estimate of the Standard Deviation of $\epsilon$, i.e. the residual. It's also known as the residual standard deviation (RSD), and it can be defined as $RSE = \sqrt{\frac{RSS}{(n-2)}}$ (e.g. see ISL page 66). $\endgroup$ Jul 23, 2017 at 21:08
  • 2
    $\begingroup$ For anyone reading the epub of ISL, you can locate "page 66" with ctrl-f "residual standard error." (Epub files do not have true page numbers). $\endgroup$ Mar 24, 2019 at 22:18
1
$\begingroup$

The residual standard error is $\sqrt{MSE}$. The $MSE$ is an unbiased estimator of $\sigma^2$, where $\sigma^2 = Var(y|x)$.

To make it more clear of the answer by @Silverfish and @Waldir Leoncio.
A summary of all definitions was shown below. Always got confused by these terms, put it here instead of making it as a comment for better formatting.

Anova table of SLR/Simple Linear Regression (DF is different for multiple regression):

Source DF Sum Sq Mean Sq F value
Regression $1$ $SSR$ $MSR = \frac{SSR}{1}$ $\frac{MSR}{MSE}$
Residual $n - 2$ $SSE$ $MSE = \frac{SSE}{n - 2}$
Total $n - 1$ $SST$

where $n$ is the sample size of $x_i$, $SST = SSE + SSR$, $SST = S_{YY} = \sum_{i = 1}^{n}{(y_i - \bar{y})^2}$, $SSE = \sum_{i = 1}^{n}{(y_i - \hat{y_i})^2}$, $SSR = \sum_{i = 1}^{n}{(\hat{y_i} - \bar{y})^2}$.

The $SSR$ is the part of variance of $y_i$ which can be explained by $\hat{y_i}$, the greater the better.

Also for SLR, $se(\beta_1) = \sqrt{MSE}/\sqrt{S_{xx}}$, where $S_{XX}$ is defined similarly as $S_{YY}$.

$\endgroup$
0
$\begingroup$

As noted by @Amelio Vazquez-Reina and @little_monster, given a (simple linear) regression model: $$ Y = \beta_0 + X \beta_1 + \epsilon $$ where $\epsilon$ is a noise term with variance $\sigma^2$, i.e. $Var(\epsilon) = \sigma^2$, Residual Standard Error ($RSE$) is an estimate of $\sigma^2$ (the latter being usually unknown). RSE is given by a formula: $$ RSE = \sqrt{\frac{RSS}{n-2}} $$ (see ISLR, page 66), where $RSS$ is a Residual Sum of Squares (sum of squared differences between the actual and predicted values): $$ RSS = \sum_{i=1}^{n}(y_i - \hat{y}_i) $$ The reason for $n-2$ in the denominator of $\sqrt{\frac{RSS}{n-2}}$ is that $\sqrt{\frac{RSS}{n}}$ would underestimate $\sigma^2$ - this is because the estimated function, $\hat{y}$, has been fit to the data such that it (by design) minimises the RSS of the residuals, but $\sigma^2$ pertains to errors (see the difference between errors and residuals). Imagine an extreme case - if there are only two data points ($n=2$), we can only fit one line - our RSS will be 0, but the true generating function (the one that was used to generate the two samples) will have a non-zero sum of errors (assuming $\sigma^2 \neq 0$). So the $n-2$ pertains to the fact that we only have $n-2$ degrees of freedom.

Remark: In general case (Multiple Linear regression with $p$ features) the $RSE$ will be estimated with: $$ RSE = \sqrt{\frac{RSS}{n-p-1}} $$ (see ISLR, page 80)

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.