Why divide by $n-2$ for residual standard errors

Question

I was just watching a lecture on statistics and someone was calculating something called the residual standard error. It looked a lot like finding the average of the square of the residuals, the residuals being the difference between the prediction of your model and the actual values. So for a linear fit, the prediction is $\hat{y}(x_i)=mx_i+b$ and the actual value is $y_i$. So the residual is $r_i = (y_i - \hat{y}(x_i))$. The residual standard error is $\frac{1}{n-2}\sum_i r_i^2$. I don't understand why dividing by $n-2$ is necessary?

Update: I have a better idea. If there are only two data points, then the residuals would all be zero. So you could not estimate the error with only two points. But this still does not explain why dividing by $n-2$ is a good idea. It only explains why the formula is undefined for $n=2$.

Answer 1

You use the residuals to estimate the distribution of the error. https://en.wikipedia.org/wiki/Errors_and_residuals , but these are different things.

• Errors are what the 'true' model includes as randomness
• Residuals are the differences that you 'observe' between a model fit and a measurement.

The residuals do not resemble the errors. When you fit a model then you will fit to the model plus the error terms. This means that the fitting has a tendency to fit a part of the error terms, in addition to the model, and this will in effect, on average, decrease the residuals in relation to the true errors.

The more parameters the model has (the more degrees of freedom the model has to fit, cover up, the partial error terms) the less the residuals will resemble the true distribution of the error.

So the expression $\frac{\sum r_i^2}{n-2}$ refers to $r_i$ as 'residual' terms, but wishes to express some idea of variance in the 'error' terms and in order to do this it need to include the '$n-2$' instead of the '$n$' term because the residual terms have a slight bias.

---

Related questions for intuition about the $n-p$ factor in ordinary linear regression

• Why don't standard estimation methods consider the distribution of the residual?

  the residuals can be considered as a projection of the errors $\mathbf{r} = \left(\mathbf{I}-\mathbf{A}(\mathbf{A}^t\mathbf{A})^{-1}\mathbf{A}^t\right) \mathbf{n}$

• Why are the residuals in $\mathbb{R}^{n-p}$?

  We can consider the variance of the observation to be split up into two orthogonal spaces, one with dimensions $p$ and the other with dimension $n-p$.

Answer 2

The issue is that the $\beta$ coefficients are estimated so as to minimize $\sum_{i=1}^{n} (y_i - \hat{y}_i)^2$ and so $n^{-1} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$ tends underestimate $\sigma^2$. This is the same reason why we often divide by $n - 1$ when estimating variances of univariate distributions. The issue is not so bad in the simple linear regression case but when $p$ becomes large the shrinkage can be substantial. For this reason we generally prefer the unbiased estimate $(n - p)^{-1} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$ because it doesn't suffer from this defect.

Answer 3

the reason why we use n-2 df instead n-1 in estimating error variance is there are two parameters estimated in each equation, we deduct 2 from the number of observations to obtain the df. This assumption underlying the Chow test which explain unbiased estimators of the true variances in the two subperiods.So, for mathematical calculation, I refer to Chow test.

Answer 4

Here's the intuitive answer.

Let's say that you need to make a regression line.

With $n=1$ data entry you can't make a line.

With $n=2$ data entries you can make exactly one line. Since you can make one and only one line you have $0=n-2$ degrees of freedom.

$\implies$With $n$ points you will have $n - 2$ degrees of freedom.

Answer 5

let's forget about the formulas and consider a simple line: y=mx+b. To calculate the slope m, we have to use two points. let's say (X1, Y1) and (X2, Y2). the slope m=(Y2-Y1)/(X2-X1). Two Y values are used to calculate the slope, which is used in calculating the Sum of Squares Error: the sum of squares of [Yi-(mx+b)]. Hence we use n-2 degrees of freedom to divide the sum of squares error to get the mean squares error. Anyway, this is a dumb answer, but it helps me get over the mysterious n-2 df.