What is residual standard error?

Question

When running a multiple regression model in R, one of the outputs is a residual standard error of 0.0589 on 95,161 degrees of freedom. I know that the 95,161 degrees of freedom is given by the difference between the number of observations in my sample and the number of variables in my model. What is the residual standard error?

Answer 1

A fitted regression model uses the parameters to generate point estimate predictions which are the means of observed responses if you were to replicate the study with the same $X$ values an infinite number of times (and when the linear model is true). The difference between these predicted values and the ones used to fit the model are called "residuals" which, when replicating the data collection process, have properties of random variables with 0 means.

The observed residuals are then used to subsequently estimate the variability in these values and to estimate the sampling distribution of the parameters. When the residual standard error is exactly 0 then the model fits the data perfectly (likely due to overfitting). If the residual standard error can not be shown to be significantly different from the variability in the unconditional response, then there is little evidence to suggest the linear model has any predictive ability.

Answer 2

Say we have the following ANOVA table (adapted from R's example(aov) command):

          Df Sum Sq Mean Sq F value Pr(>F)
Model      1   37.0   37.00   0.483  0.525
Residuals  4  306.3   76.57               

If you divide the sum of squares from any source of variation (model or residuals) by its respective degrees of freedom, you get the mean square. Particularly for the residuals:

$$ \frac{306.3}{4} = 76.575 \approx 76.57 $$

So 76.57 is the mean square of the residuals, i.e., the amount of residual (after applying the model) variation on your response variable.

The residual standard error you've asked about is nothing more than the positive square root of the mean square error. In my example, the residual standard error would be equal to $\sqrt{76.57}$, or approximately 8.75. R would output this information as "8.75 on 4 degrees of freedom".

Answer 3

Typically you will have a regression model looks like this: $$ Y = \beta_{0} + \beta_{1}X + \epsilon $$ where $ \epsilon $ is an error term independent of $ X $.

If $ \beta_{0} $ and $ \beta_{1} $ are known, we still cannot perfectly predict Y using X due to $ \epsilon $. Therefore, we use RSE as a judgement value of the Standard Deviation of $ \epsilon $.

RSE is explained pretty much clearly in "Introduction to Statistical Learning".

Answer 4

The residual standard error is $\sqrt{MSE}$. The $MSE$ is an unbiased estimator of $\sigma^2$, where $\sigma^2 = Var(y|x)$.

To make it more clear of the answer by @Silverfish and @Waldir Leoncio.
A summary of all definitions was shown below. Always got confused by these terms, put it here instead of making it as a comment for better formatting.

Anova table of SLR/Simple Linear Regression (DF is different for multiple regression):

Source	DF	Sum Sq	Mean Sq	F value
Regression	$1$	$SSR$	$MSR = \frac{SSR}{1}$	$\frac{MSR}{MSE}$
Residual	$n - 2$	$SSE$	$MSE = \frac{SSE}{n - 2}$
Total	$n - 1$	$SST$

where $n$ is the sample size of $x_i$, $SST = SSE + SSR$, $SST = S_{YY} = \sum_{i = 1}^{n}{(y_i - \bar{y})^2}$, $SSE = \sum_{i = 1}^{n}{(y_i - \hat{y_i})^2}$, $SSR = \sum_{i = 1}^{n}{(\hat{y_i} - \bar{y})^2}$.

The $SSR$ is the part of variance of $y_i$ which can be explained by $\hat{y_i}$, the greater the better.

Also for SLR, $se(\beta_1) = \sqrt{MSE}/\sqrt{S_{xx}}$, where $S_{XX}$ is defined similarly as $S_{YY}$.

Answer 5

As noted by @Amelio Vazquez-Reina and @little_monster, given a (simple linear) regression model: $$ Y = \beta_0 + X \beta_1 + \epsilon $$ where $\epsilon$ is a noise term with variance $\sigma^2$, i.e. $Var(\epsilon) = \sigma^2$, Residual Standard Error ($RSE$) is an estimate of $\sigma^2$ (the latter being usually unknown). RSE is given by a formula: $$ RSE = \sqrt{\frac{RSS}{n-2}} $$ (see ISLR, page 66), where $RSS$ is a Residual Sum of Squares (sum of squared differences between the actual and predicted values): $$ RSS = \sum_{i=1}^{n}(y_i - \hat{y}_i) $$ The reason for $n-2$ in the denominator of $\sqrt{\frac{RSS}{n-2}}$ is that $\sqrt{\frac{RSS}{n}}$ would underestimate $\sigma^2$ - this is because the estimated function, $\hat{y}$, has been fit to the data such that it (by design) minimises the RSS of the residuals, but $\sigma^2$ pertains to errors (see the difference between errors and residuals). Imagine an extreme case - if there are only two data points ($n=2$), we can only fit one line - our RSS will be 0, but the true generating function (the one that was used to generate the two samples) will have a non-zero sum of errors (assuming $\sigma^2 \neq 0$). So the $n-2$ pertains to the fact that we only have $n-2$ degrees of freedom.

Remark: In general case (Multiple Linear regression with $p$ features) the $RSE$ will be estimated with: $$ RSE = \sqrt{\frac{RSS}{n-p-1}} $$ (see ISLR, page 80)