# Why is RSS distributed chi square times n-p?

I would like to understand why, under the OLS model, the RSS (residual sum of squares) is distributed $$\chi^2\cdot (n-p)$$ ($p$ being the number of parameters in the model, $n$ the number of observations).

I apologize for asking such a basic question, but I seem to not be able to find the answer online (or in my, more application oriented, textbooks).

• Note that the answers demonstrate the assertion is not quite right: the distribution of RSS is $\sigma^2$ (not $n-p$) times a $\chi^2(n-p)$ distribution where $\sigma^2$ is the true variance of the errors. – whuber Nov 21 '13 at 19:40

## 2 Answers

I consider the following linear model: ${y} = X \beta + \epsilon$.

The vector of residuals is estimated by

$$\hat{\epsilon} = y - X \hat{\beta} = (I - X (X'X)^{-1} X') y = Q y = Q (X \beta + \epsilon) = Q \epsilon$$

where $Q = I - X (X'X)^{-1} X'$.

Observe that $\textrm{tr}(Q) = n - p$ (the trace is invariant under cyclic permutation) and that $Q'=Q=Q^2$. The eigenvalues of $Q$ are therefore $0$ and $1$ (some details below). Hence, there exists a unitary matrix $V$ such that (matrices are diagonalizable by unitary matrices if and only if they are normal.)

$$V'QV = \Delta = \textrm{diag}(\underbrace{1, \ldots, 1}_{n-p \textrm{ times}}, \underbrace{0, \ldots, 0}_{p \textrm{ times}})$$

Now, let $K = V' \hat{\epsilon}$.

Since $\hat{\epsilon} \sim N(0, \sigma^2 Q)$, we have $K \sim N(0, \sigma^2 \Delta)$ and therefore $K_{n-p+1}=\ldots=K_n=0$. Thus

$$\frac{\|K\|^2}{\sigma^2} = \frac{\|K^{\star}\|^2}{\sigma^2} \sim \chi^2_{n-p}$$

with $K^{\star} = (K_1, \ldots, K_{n-p})'$.

Further, as $V$ is a unitary matrix, we also have

$$\|\hat{\epsilon}\|^2 = \|K\|^2=\|K^{\star}\|^2$$

Thus

$$\frac{\textrm{RSS}}{\sigma^2} \sim \chi^2_{n-p}$$

Finally, observe that this result implies that

$$E\left(\frac{\textrm{RSS}}{n-p}\right) = \sigma^2$$

Since $Q^2 - Q =0$, the minimal polynomial of $Q$ divides the polynomial $z^2 - z$. So, the eigenvalues of $Q$ are among $0$ and $1$. Since $\textrm{tr}(Q) = n-p$ is also the sum of the eigenvalues multiplied by their multiplicity, we necessarily have that $1$ is an eigenvalue with multiplicity $n-p$ and zero is an eigenvalue with multiplicity $p$.

• (+1) Good answer. One can restrict attention to orthogonal, instead of unitary, $V$ since $Q$ is real and symmetric. Also, what is $\mathrm{SCR}$? I do not see it defined. By slightly rejiggering the argument, one can also avoid the use of a degenerate normal, in case that causes some consternation to those not familiar with it. – cardinal Dec 25 '11 at 17:29
• @Cardinal. Good point. SCR ('Somme des Carrés Résiduels' in french) should have been RSS. – ocram Dec 25 '11 at 17:53
• Thank you for the detailed answer Ocram! Some steps will require me to look more, but I have an outline to think about now - thanks! – Tal Galili Dec 25 '11 at 21:45
• @Glen_b: Oh, I made an edit a couple of days ago to change SCR to SRR. I didn't remember that SCR is mentionned in my comment. Sorry for the confusion. – ocram Nov 18 '13 at 6:03
• @Glen_b: It was supposed to mean RSS :-S Edited again. Thx – ocram Nov 18 '13 at 6:15

IMHO, the matricial notation $Y=X\beta+\epsilon$ complicates things. Pure vector space language is cleaner. The model can be written $\boxed{Y=\mu + \sigma G}$ where $G$ has the standard normal distributon on $\mathbb{R}^n$ and $\mu$ is assumed to belong to a vector subspace $W \subset \mathbb{R}^n$.

Now the language of elementary geometry comes into play. The least-squares estimator $\hat\mu$ of $\mu$ is nothing but $P_WY$: the orthogonal projection of the observable $Y$ on the space $W$ to which $\mu$ is assumed to belong. The vector of residuals is $P^\perp_WY$: projection on the orthogonal complement $W^\perp$ of $W$ in $\mathbb{R^n}$. The dimension of $W^\perp$ is $\dim(W^\perp)=n-\dim(W)$.

Finally, $$P^\perp_WY = P^\perp_W(\mu + \sigma G) = 0 + \sigma P^\perp_WG,$$ and $P^\perp_WG$ has the standard normal distribution on $W^\perp$, hence its squared norm has the $\chi^2$ distribution with $\dim(W^\perp)$ degrees of freedom.

This demonstration uses only one theorem, actually a definition-theorem:

Definition and theorem. A random vector in $\mathbb{R}^n$ has the standard normal distribution on a vector space $U \subset \mathbb{R}^n$ if it takes its values in $U$ and its coordinates in one ($\iff$ in all) orthonormal basis of $U$ are independent one-dimensional standard normal distributions

(from this definition-theorem, Cochran's theorem is so obvious that it is not worth to state it)