$$\text{RSE}=\sqrt{\frac{1}{n-2}\text{RSS}}=\sqrt{\frac{1}{n-2}\sum_{i=1}^n (y_i-\hat{y_i})^2}$$

Context: Simple Linear Regression, an intercept and a slope

I have 2 question regarding this issue.

  1. Why should we divide RSS ?
  2. Why the divisor should be $n-2$, not $n$ or $n-1$ ?
  • 1
    $\begingroup$ What is the context? For example, are you doing simple linear regression? $\endgroup$ – Glen_b Mar 29 '16 at 8:36
  • $\begingroup$ @Glen_b: Yes, simple linear regression. an intercept and a slope. $\endgroup$ – Jul Mar 30 '16 at 7:02
  • $\begingroup$ Please edit the information into your question $\endgroup$ – Glen_b Mar 30 '16 at 7:14
  • $\begingroup$ @Glen_b: Done... $\endgroup$ – Jul Mar 30 '16 at 19:17
  • 1
    $\begingroup$ Because it's perfectly valid to divide by $n$ (that gives the Maximum Likelihood estimator of the variance of the errors) or by other factors that asymptotically approach $n$ (they will give various Bayesian posterior estimates), any valid answer must appeal to some desirable property of this Ordinary Least Squares estimate: namely, that it produces an unbiased estimator of the variance. $\endgroup$ – whuber Aug 12 '18 at 13:50

The reason is based on trying to get an unbiased estimator of the underlying error variance in the regression. In a simple linear regression with normal error terms it can be shown that:

$$\text{RSS}(\mathbf{x},\mathbf{Y}) \equiv \sum_{i=1}^n (Y_i - \hat{Y}_i) \sim \sigma^2 \cdot \text{Chi-Sq}(df = n-2).$$

That is, under the standard assumption of normally distributed errors, the residual sum-of-squares has a chi-squared distribution with $n-2$ degrees of freedom. (This is called the residual degrees-of-freedom.) One consequence of this distributional result is that the residual sum-of-squares has expected value $\mathbb{E} ( \text{RSS}(\mathbf{x},\mathbf{Y}) ) = \sigma^2 (n-2)$. You can see from this result that the residual sum-of-squares will tend to be larger for larger data sets (i.e., it is an increasing function of $n$) and it is not a useful estimator of the error variance.

Unbiased estimation of the error variance: To get an unbiased estimator of the error variance, we divide by the residual degrees-of-freedom to get the residual mean-square:

$$\text{RMS}(\mathbf{x},\mathbf{Y}) \equiv \frac{\text{RSS}(\mathbf{x},\mathbf{Y})}{n-2} \sim \sigma^2 \cdot \frac{\text{Chi-Sq}(df = n-2)}{n-2}.$$

This statistic has expected value $\mathbb{E} ( \text{RMS}(\mathbf{x},\mathbf{Y}) ) = \sigma^2$, so it gives an unbiased estimator for the error variance in the regression. The corresponding statistic $\text{RME} = \sqrt{\text{RMS}}$ gives an estimator for $\sigma$, which is the standard deviation of the error term. (Note that the latter is not unbiased, since unbiased estimation of the variance leads to biased estimation of the standard deviation.)

Extension to multiple regression: This result is easily extended to multiple regression (with an intercept term and $m$ explanatory variables) where we have:

$$\text{RSS}(\mathbf{x},\mathbf{Y}) \equiv \sum_{i=1}^n (Y_i - \hat{Y}_i) \sim \sigma^2 \cdot \text{Chi-Sq}(df = n-m-1).$$

In this case the regression mean-square (which estimates the error variance) is:

$$\text{RMS}(\mathbf{x},\mathbf{Y}) \equiv \frac{\text{RSS}(\mathbf{x},\mathbf{Y})}{n-m-1} \sim \sigma^2 \cdot \frac{\text{Chi-Sq}(df = n-m-1)}{n-m-1}.$$

Proof of this general distributional result relies on material relating to quadratic forms of normal distributions, which is beyond the level of mathematics that is usually presented in introductory statistics courses. For information on the derivation of these results you can consult an advanced text on linear regression.

  • $\begingroup$ can you kindly direct to the proof of relation between RSS and chi square having n-2 df? $\endgroup$ – Parthiban Rajendran Nov 22 '18 at 12:24
  • $\begingroup$ This would be found in most advanced texts on regression analysis, and also perhaps in books on quadratic forms of normal random vectors. You can find a proof on this site here. $\endgroup$ – Ben Nov 22 '18 at 23:02

In linear regression, if you are observing the relationship between a single predictor and its response then the equation is of the form

$$Y = b_0 + b_1 X.$$

Here, $Y$ is the response variable and $X$ is the predictor variable; $b_1$ and $b_0$ are coefficients that need to be found. Now we have two values to be found and so our degrees of freedom are $n-2$.

Degree of freedom is the freedom of selecting a value e.g. if you want to wear a different tie everyday and you have a total of 7 ties then you have the freedom to choose any tie on the first day, but then this freedom will decrease everyday till last day, when you can't choose a tie and have no freedom to choose it.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.