14
$\begingroup$

I was just watching a lecture on statistics and someone was calculating something called the residual standard error. It looked a lot like finding the average of the square of the residuals, the residuals being the difference between the prediction of your model and the actual values. So for a linear fit, the prediction is $\hat{y}(x_i)=mx_i+b$ and the actual value is $y_i$. So the residual is $r_i = (y_i - \hat{y}(x_i))$. The residual standard error is $\frac{1}{n-2}\sum_i r_i^2$. I don't understand why dividing by $n-2$ is necessary?

Update: I have a better idea. If there are only two data points, then the residuals would all be zero. So you could not estimate the error with only two points. But this still does not explain why dividing by $n-2$ is a good idea. It only explains why the formula is undefined for $n=2$.

$\endgroup$
2
  • 5
    $\begingroup$ "necessary" isn't really the right word. There are reasons you might prefer to do it (over just dividing by $n$ say), but you don't have to do it. $\endgroup$
    – Glen_b
    Commented Mar 13, 2016 at 9:26
  • 2
    $\begingroup$ See stats.stackexchange.com/a/76748/77222 for a rigorous simple answer. $\endgroup$ Commented Apr 9, 2019 at 21:05

5 Answers 5

10
$\begingroup$

You use the residuals to estimate the distribution of the error. https://en.wikipedia.org/wiki/Errors_and_residuals , but these are different things.

  • Errors are what the 'true' model includes as randomness
  • Residuals are the differences that you 'observe' between a model fit and a measurement.

The residuals do not resemble the errors. When you fit a model then you will fit to the model plus the error terms. This means that the fitting has a tendency to fit a part of the error terms, in addition to the model, and this will in effect, on average, decrease the residuals in relation to the true errors.

The more parameters the model has (the more degrees of freedom the model has to fit, cover up, the partial error terms) the less the residuals will resemble the true distribution of the error.

So the expression $\frac{\sum r_i^2}{n-2}$ refers to $r_i$ as 'residual' terms, but wishes to express some idea of variance in the 'error' terms and in order to do this it need to include the '$n-2$' instead of the '$n$' term because the residual terms have a slight bias.


Related questions for intuition about the $n-p$ factor in ordinary linear regression

$\endgroup$
3
  • 3
    $\begingroup$ I really liked your explanation. Quite clear. $\endgroup$ Commented Feb 16, 2018 at 15:29
  • $\begingroup$ Is this similar in spirit to Bessel's correction for sample standard deviations? $\endgroup$ Commented Jul 10, 2021 at 19:50
  • $\begingroup$ The spirit of this is right, but the letter is a little erratic. The assertion that residual is less than error seems to need a qualification "typically" and a qualification that you mean in absolute value. Also, you seem to equivocate between individual residuals or errors (positive, zero or negative) and e.g. residuals as meaning sum of squared residuals. $\endgroup$
    – Nick Cox
    Commented Jun 17, 2023 at 9:13
6
$\begingroup$

The issue is that the $\beta$ coefficients are estimated so as to minimize $\sum_{i=1}^{n} (y_i - \hat{y}_i)^2$ and so $n^{-1} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$ tends underestimate $\sigma^2$. This is the same reason why we often divide by $n - 1$ when estimating variances of univariate distributions. The issue is not so bad in the simple linear regression case but when $p$ becomes large the shrinkage can be substantial. For this reason we generally prefer the unbiased estimate $(n - p)^{-1} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$ because it doesn't suffer from this defect.

$\endgroup$
3
$\begingroup$

the reason why we use n-2 df instead n-1 in estimating error variance is there are two parameters estimated in each equation, we deduct 2 from the number of observations to obtain the df. This assumption underlying the Chow test which explain unbiased estimators of the true variances in the two subperiods.So, for mathematical calculation, I refer to Chow test.

$\endgroup$
1
  • 2
    $\begingroup$ The "two parameter" idea is not an explanation; it is only a heuristic (to help people remember what denominator to use). An explanation would offer reasons why the value 2 is subtracted from $n$ in this particular formula. BTW, the value to subtract rarely is the number of parameters in the regression model: at a minimum, at least one parameter associated with the variance of the error terms is estimated as well, but it is not counted in this particular heuristic. $\endgroup$
    – whuber
    Commented Feb 16, 2018 at 14:59
2
$\begingroup$

Here's the intuitive answer.

Let's say that you need to make a regression line.

With $n=1$ data entry you can't make a line.

With $n=2$ data entries you can make exactly one line. Since you can make one and only one line you have $0=n-2$ degrees of freedom.

$\implies$With $n$ points you will have $n - 2$ degrees of freedom.

$\endgroup$
2
  • 4
    $\begingroup$ Although the counting is correct, it remains to show why it might be relevant: that's what the last part of the question is asking about. $\endgroup$
    – whuber
    Commented Apr 9, 2019 at 21:12
  • 2
    $\begingroup$ I think the answer provides some intuition with respect to the degrees of freedom. However, the example mentioned is limited to a linear regression line. $\endgroup$ Commented Nov 27, 2019 at 17:18
0
$\begingroup$

let's forget about the formulas and consider a simple line: y=mx+b. To calculate the slope m, we have to use two points. let's say (X1, Y1) and (X2, Y2). the slope m=(Y2-Y1)/(X2-X1). Two Y values are used to calculate the slope, which is used in calculating the Sum of Squares Error: the sum of squares of [Yi-(mx+b)]. Hence we use n-2 degrees of freedom to divide the sum of squares error to get the mean squares error. Anyway, this is a dumb answer, but it helps me get over the mysterious n-2 df.

$\endgroup$
2
  • 1
    $\begingroup$ As it’s currently written, your answer is unclear. Please edit to add additional details that will help others understand how this addresses the question asked. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Commented Aug 29, 2023 at 8:43
  • $\begingroup$ As the regression line need not go through any data points at all, the idea of fixing it by choosing two data points is incorrect as well as irrelevant, so far as I can see. . $\endgroup$
    – Nick Cox
    Commented Aug 29, 2023 at 9:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.