2
$\begingroup$

This might be trivial and vague question, but I still don't understand why when creating test statistics or estimators we always divide by the degree of freedom. Just to give examples of what I'm talking about:

The F-test for linear regression for example: $F=(TSS-RSS/p)/(RSS/n-p-1)$ or with the residual standard error $RSE=sqrt(RSS/n-p-1)$ RSS is the sum of residuals squared.

Is it a way to make this statistics approachable by known statistics such as t and F statistics for example?

$\endgroup$
6
$\begingroup$

Here is an argument for why we divide by degrees of freedom, in a simple case. Let $X_1, X_2, \cdots X_n$ be independent and identically distributed with mean $\mu$ and variance $\sigma^2$. Consider the sample variance as an estimator for $\sigma^2$. $$S^2 = \frac{1}{n-1}\sum_{i=1}^n(X_i - \bar{X})^2 = \frac{1}{n-1}\left(\sum_{i=1}^nX_i^2 - n\bar{X}^2\right)$$ We can show that $S^2$ is unbiased for $\sigma^2$. \begin{align*} E(S^2) &= \frac{1}{n-1}\left(E\left(\sum X_i^2\right) - nE(\bar X^2)\right) \\ &= \frac{1}{n-1}(nE(X_i^2) - nE(\bar X^2)) \\ &= \frac{1}{n-1}(n(\mu^2 + \sigma^2) - n(\mu^2 + \sigma^2/n)) \\ &= \frac{1}{n-1}(n\sigma^2 - \sigma^2) = \sigma^2 \end{align*} Note that if we had divided by anything other than $n-1$, this estimator would be biased. In fact, it can be shown that $S^2$ has uniformly minimum variance of all unbiased estimators of $\sigma^2$ in many cases.

On the other hand, assume that $\mu$ is known (not very useful in practice). Now the estimator $$\hat\sigma^2 = \frac{1}{n}\sum_{i=1}^n(X_i-\mu)^2$$ is unbiased. This is justified, since we are not losing a degree of freedom to estimate $\bar{X}$.

This is just one simple case where dividing by degrees of freedom makes sense, but the justification works for other statistics as well.

$\endgroup$
  • $\begingroup$ Thanks for your answer. So, if I understand then the purpose is to guarantee that the estimator used in the given problem is the best there is? And this is done by making it unbiased and minimizing its standard error. $\endgroup$ – Youssef Esseddiq Nov 14 '18 at 8:19
  • 2
    $\begingroup$ The example of $F$ ratio statistics in the question indicates the answer has nothing to do with bias. $\endgroup$ – whuber Nov 14 '18 at 15:44
  • $\begingroup$ @whuber, I understand your point, and agree this isn't a complete answer. The OPs language indicates that he is interested in why we divide by degrees of freedom in general. My example is only intended to show that it is a reasonable thing to do, in a case that nearly everybody is familiar with. Many of my students find this example easy to understand, and therefore helpful. $\endgroup$ – knrumsey Nov 14 '18 at 18:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.