I would like to understand why, under the OLS model, the RSS (residual sum of squares) is distributed $$\chi^2\cdot (n-p)$$ ($p$ being the number of parameters in the model, $n$ the number of observations).

I apologize for asking such a basic question, but I seem to not be able to find the answer online (or in my, more application oriented, textbooks).

  • 5
    $\begingroup$ Note that the answers demonstrate the assertion is not quite right: the distribution of RSS is $\sigma^2$ (not $n-p$) times a $\chi^2(n-p)$ distribution where $\sigma^2$ is the true variance of the errors. $\endgroup$ – whuber Nov 21 '13 at 19:40

I consider the following linear model: ${y} = X \beta + \epsilon$.

The vector of residuals is estimated by

$$\hat{\epsilon} = y - X \hat{\beta} = (I - X (X'X)^{-1} X') y = Q y = Q (X \beta + \epsilon) = Q \epsilon$$

where $Q = I - X (X'X)^{-1} X'$.

Observe that $\textrm{tr}(Q) = n - p$ (the trace is invariant under cyclic permutation) and that $Q'=Q=Q^2$. The eigenvalues of $Q$ are therefore $0$ and $1$ (some details below). Hence, there exists a unitary matrix $V$ such that (matrices are diagonalizable by unitary matrices if and only if they are normal.)

$$V'QV = \Delta = \textrm{diag}(\underbrace{1, \ldots, 1}_{n-p \textrm{ times}}, \underbrace{0, \ldots, 0}_{p \textrm{ times}})$$

Now, let $K = V' \hat{\epsilon}$.

Since $\hat{\epsilon} \sim N(0, \sigma^2 Q)$, we have $K \sim N(0, \sigma^2 \Delta)$ and therefore $K_{n-p+1}=\ldots=K_n=0$. Thus

$$\frac{\|K\|^2}{\sigma^2} = \frac{\|K^{\star}\|^2}{\sigma^2} \sim \chi^2_{n-p}$$

with $K^{\star} = (K_1, \ldots, K_{n-p})'$.

Further, as $V$ is a unitary matrix, we also have

$$\|\hat{\epsilon}\|^2 = \|K\|^2=\|K^{\star}\|^2$$


$$\frac{\textrm{RSS}}{\sigma^2} \sim \chi^2_{n-p}$$

Finally, observe that this result implies that

$$E\left(\frac{\textrm{RSS}}{n-p}\right) = \sigma^2$$

Since $Q^2 - Q =0$, the minimal polynomial of $Q$ divides the polynomial $z^2 - z$. So, the eigenvalues of $Q$ are among $0$ and $1$. Since $\textrm{tr}(Q) = n-p$ is also the sum of the eigenvalues multiplied by their multiplicity, we necessarily have that $1$ is an eigenvalue with multiplicity $n-p$ and zero is an eigenvalue with multiplicity $p$.

  • 1
    $\begingroup$ (+1) Good answer. One can restrict attention to orthogonal, instead of unitary, $V$ since $Q$ is real and symmetric. Also, what is $\mathrm{SCR}$? I do not see it defined. By slightly rejiggering the argument, one can also avoid the use of a degenerate normal, in case that causes some consternation to those not familiar with it. $\endgroup$ – cardinal Dec 25 '11 at 17:29
  • 2
    $\begingroup$ @Cardinal. Good point. SCR ('Somme des Carrés Résiduels' in french) should have been RSS. $\endgroup$ – ocram Dec 25 '11 at 17:53
  • $\begingroup$ Thank you for the detailed answer Ocram! Some steps will require me to look more, but I have an outline to think about now - thanks! $\endgroup$ – Tal Galili Dec 25 '11 at 21:45
  • $\begingroup$ @Glen_b: Oh, I made an edit a couple of days ago to change SCR to SRR. I didn't remember that SCR is mentionned in my comment. Sorry for the confusion. $\endgroup$ – ocram Nov 18 '13 at 6:03
  • $\begingroup$ @Glen_b: It was supposed to mean RSS :-S Edited again. Thx $\endgroup$ – ocram Nov 18 '13 at 6:15

IMHO, the matricial notation $Y=X\beta+\epsilon$ complicates things. Pure vector space language is cleaner. The model can be written $\boxed{Y=\mu + \sigma G}$ where $G$ has the standard normal distributon on $\mathbb{R}^n$ and $\mu$ is assumed to belong to a vector subspace $W \subset \mathbb{R}^n$.

Now the language of elementary geometry comes into play. The least-squares estimator $\hat\mu$ of $\mu$ is nothing but $P_WY$: the orthogonal projection of the observable $Y$ on the space $W$ to which $\mu$ is assumed to belong. The vector of residuals is $P^\perp_WY$: projection on the orthogonal complement $W^\perp$ of $W$ in $\mathbb{R^n}$. The dimension of $W^\perp$ is $\dim(W^\perp)=n-\dim(W)$.

Finally, $$P^\perp_WY = P^\perp_W(\mu + \sigma G) = 0 + \sigma P^\perp_WG,$$ and $P^\perp_WG$ has the standard normal distribution on $W^\perp$, hence its squared norm has the $\chi^2$ distribution with $\dim(W^\perp)$ degrees of freedom.

This demonstration uses only one theorem, actually a definition-theorem:

Definition and theorem. A random vector in $\mathbb{R}^n$ has the standard normal distribution on a vector space $U \subset \mathbb{R}^n$ if it takes its values in $U$ and its coordinates in one ($\iff$ in all) orthonormal basis of $U$ are independent one-dimensional standard normal distributions

(from this definition-theorem, Cochran's theorem is so obvious that it is not worth to state it)


Once we've established that $\hat\epsilon=(I-H)\epsilon$, we can apply the following lemma:

Lemma: If $A_{n\times n}$ is a symmetric and idempotent real matrix, then there exists a matrix $U$ with orthonormal columns such that $A=UU^T$. The matrix $U$ is $n\times r$, where $r$ equals the rank of $A$.

Proof: The spectral theorem for symmetric matrices asserts $A=UDU^T$ where $D_{n\times n}$ is a diagonal matrix of the eigenvalues $\lambda_1,\ldots,\lambda_n$ of $A$ and $U_{n\times n}$ is an orthogonal matrix whose columns are the corresponding eigenvectors $u_1,\ldots,u_n$. Since $A$ is idempotent, each eigenvalue is either zero or one (reason: $Au=\lambda u$ implies $\lambda u =Au=A(Au)=A\lambda u=\lambda^2u$). Delete from $U$ the columns corresponding to zero eigenvalue, leaving an $n\times r$ matrix; the diagonal matrix $D$ becomes the identity. To determine $r$, note that the columns remaining in $U$ each satisfy $Au_i=u_i$, hence they form a basis for the range of $A$; so $\operatorname {rank}(A)=r$.

Applying the lemma, write $I-H=UU^T$ where $U_{n\times r}$ has orthonormal columns and $r=\operatorname{rank}(I-H)$. Then $\hat\epsilon:=(I-H)\epsilon=U(U^T\epsilon) $. Observe that $N:=U^T\epsilon$ is an $r$-dimensional random vector having multivariate normal distribution with mean zero and covariance matrix $$ \operatorname{Var}(N)=E(U^T\epsilon)(U^T\epsilon)^T=U^TE(\epsilon\epsilon^T)U=\sigma^2 (U^TU)=\sigma^2I_{r\times r} $$ and that $$ \operatorname{RSS}:=\hat\epsilon^T\hat\epsilon=(UN)^T(UN)=N^T(U^TU)N=N^TN. $$ Conclude $\operatorname{RSS}/\sigma^2$ is the sum of squares of $r$ IID standard normal variables and therefore has chi-square($r$) distribution.

To finish, we find $r=n-\operatorname{rank}(X)$: Consider for $v\in{\mathbb R}^n$ the decomposition $v=Hv + (I-H)v$. Idempotency of $H$ implies $(Hv)^T(I-H)v'=0$ for all $v,v'$, whence ${\mathbb R}^n$ is the direct sum of the subspaces $\operatorname{range}(H)$ and $\operatorname{range}(I-H)$, and so $$n=\operatorname{dim}\operatorname{range}(H) + \operatorname{dim}\operatorname{range}(I-H) =\operatorname{rank}(H) + \operatorname{rank}(I-H) = \operatorname{rank}(X)+r.$$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.