4
$\begingroup$

I was just watching a lecture on statistics and someone was calculating something called the residual standard error. It looked a lot like finding the average of the square of the residuals, the residuals being the difference between the prediction of your model and the actual values. So for a linear fit, the prediction is $\hat{y}(x_i)=mx_i+b$ and the actual value is $y_i$. So the residual is $r_i = (y_i - \hat{y}(x_i))$. The residual standard error is $\frac{1}{n-2}\sum_i r_i^2$. I don't understand why dividing by $n-2$ is necessary?

Update: I have a better idea. If there are only two data points, then the residuals would all be zero. So you could not estimate the error with only two points. But this still does not explain why dividing by $n-2$ is a good idea. It only explains why the formula is undefined for $n=2$.

$\endgroup$
  • 3
    $\begingroup$ "necessary" isn't really the right word. There are reasons you might prefer to do it (over just dividing by $n$ say), but you don't have to do it. $\endgroup$ – Glen_b Mar 13 '16 at 9:26
  • 1
    $\begingroup$ See stats.stackexchange.com/a/76748/77222 for a rigorous simple answer. $\endgroup$ – Jarle Tufto Apr 9 '19 at 21:05
4
$\begingroup$

You use the residuals to estimate the distribution of the error. https://en.wikipedia.org/wiki/Errors_and_residuals , but these are different things.

  • Error's are what the 'true' model includes as randomness
  • Residuals are the differentiations that you 'observe' between a model fit and a measurement.

The residuals do not resemble the errors. When you fit a model then you will fit to the model plus the error terms. This means that the fitting has a tendency to fit a part of the error terms, in addition to the model, and this will in effect decrease the residuals in relation to the true errors (ie residuals < error, and in this particular case $residuals = error/(n-2)$).

The more parameters the model has (the more degrees of freedom the model has to fit, cover up, the partial error terms) the less the residuals will resemble the true distribution of the error.

So the expression $\frac{\sum r_i^2}{n-2}$ refers to $r_i$ as 'residiual' terms, but wishes to express some idea of variance in the 'error' terms and in order to do this it need to include the '$n-2$' instead of the '$n$' term because the residual terms have a slight bias.

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ I really liked your explanation. Quite clear. $\endgroup$ – Alexey Burnakov Feb 16 '18 at 15:29
3
$\begingroup$

The issue is that the $\beta$ coefficients are estimated so as to minimize $\sum_{i=1}^{n} (y_i - \hat{y}_i)^2$ and so $n^{-1} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$ tends underestimate $\sigma^2$. This is the same reason why we often divide by $n - 1$ when estimating variances of univariate distributions. The issue is not so bad in the simple linear regression case but when $p$ becomes large the shrinkage can be substantial. For this reason we generally prefer the unbiased estimate $(n - p)^{-1} \sum_{i=1}^{n} (y_i - \hat{y}_i)^2$ because it doesn't suffer from this defect.

| cite | improve this answer | |
$\endgroup$
2
$\begingroup$

the reason why we use n-2 df instead n-1 in estimating error variance is there are two parameters estimated in each equation, we deduct 2 from the number of observations to obtain the df. This assumption underlying the Chow test which explain unbiased estimators of the true variances in the two subperiods.So, for mathematical calculation, I refer to Chow test.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ The "two parameter" idea is not an explanation; it is only a heuristic (to help people remember what denominator to use). An explanation would offer reasons why the value 2 is subtracted from $n$ in this particular formula. BTW, the value to subtract rarely is the number of parameters in the regression model: at a minimum, at least one parameter associated with the variance of the error terms is estimated as well, but it is not counted in this particular heuristic. $\endgroup$ – whuber Feb 16 '18 at 14:59
1
$\begingroup$

Here's the intuitive answer.

Let's say that you need to make a regression line.

With $n=1$ data entry you can't make a line.

With $n=2$ data entries you can make exactly one line. Since you can make one and only one line you have $0=n-2$ degrees of freedom.

$\implies$With $n$ points you will have $n - 2$ degrees of freedom.

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ Although the counting is correct, it remains to show why it might be relevant: that's what the last part of the question is asking about. $\endgroup$ – whuber Apr 9 '19 at 21:12
  • 1
    $\begingroup$ I think the answer provides some intuition with respect to the degrees of freedom. However, the example mentioned is limited to a linear regression line. $\endgroup$ – Nadia Merquez Nov 27 '19 at 17:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.