# Distribution of sum of squares error for linear regression?

I know that distribution of sample variance
$$\sum\frac{(X_i-\bar{X})^2}{\sigma^2}\sim \chi^2_{(n-1)}$$ $$\sum\frac{(X_i-\bar{X})^2}{n-1}\sim \frac{\sigma^2}{n-1}\chi^2_{(n-1)}$$ It's from the fact that $(X-\bar{X})^2$ can be expressed in matrix form, $xAx'$ (where A: symmetric), and it could be again be expressed in: $x'QDQ'x$ (where Q: orthonormal, D:diagonal matrix).

What about $\sum(Y_i-\hat{\beta}_0-\hat{\beta}_1X_i)^2$, given the assumption $(Y - \beta_0 - \beta_1X)\sim \mathcal{N}(0, \sigma^2)$ ?

I figure $$\sum\frac{(Y_i-\hat{\beta}_0-\hat{\beta}_1X_i)^2}{\sigma^2}\sim \chi^2_{(n-2)}.$$

But I have no clue how to prove it or show it.

Is it distributed exactly as $\chi^2_{(n-2)}$?

• Is this homework? If so, please use the Homework tag. – MånsT Mar 20 '12 at 9:23
• No, it's not. I think it is true bcoz after all, sum of squares is a square of linear combination of Y's given constant X's. But is it? Simple proof like this would be appreciated!math.stackexchange.com/questions/47009/… – KH Kim Mar 20 '12 at 10:32
• The descriptions you give in both the question and your comment are a little bit muddled. Have you written out what your matrix $A$ must be for the sample variance? Does that help you see how to generalize? – cardinal Mar 20 '12 at 13:22
• Corrected for D. I think the critical point is that diagonal element of D should be something like (1,1,1,...,1,0,0). Is there any way to prove it? or Is there anyway to show that $\chi^2(n)=\chi^2(n-2)+\chi^2(1)+\chi^2(1)$ where sse/$\sigma^2 \sim \chi^2(n-2)$, $\sum{e_i^2}/\sigma^2 \sim \chi^2(n)$ – KH Kim Mar 20 '12 at 14:09
• stats.stackexchange.com/q/20227/119261 – StubbornAtom Jun 18 at 18:41

We can prove this for more general case of $p$ variables by using the "hat matrix" and some of its useful properties. These results are usually much harder to state in non matrix terms because of the use of the spectral decomposition.

Now in matrix version of least squares, the hat matrix is $H=X(X^TX)^{-1}X^T$ where $X$ has $n$ rows and $p+1$ columns (column of ones for $\beta_0$). Assume full column rank for convenience - else you could replace $p+1$ by the column rank of $X$ in the following. We can write the fitted values as $\hat{Y}_i=\sum_{j=1}^nH_{ij}Y_j$ or in matrix notation $\hat{Y}=HY$. Using this, we can write the sum of squares as:

$$\frac{\sum_{i=1}(Y-\hat{Y_i})^2}{\sigma^2}=\frac{(Y-\hat{Y})^T(Y-\hat{Y})}{\sigma^2}=\frac{(Y-HY)^T(Y-HY)}{\sigma^2}$$ $$=\frac{Y^T(I_n-H)Y}{\sigma^2}$$

Where $I_n$ is an identity matrix of order $n$. The last step follows from the fact that $H$ is an idepotent matrix, as $$H^2=[X(X^TX)^{-1}X^T][X(X^TX)^{-1}X^T]=X(X^TX)^{-1}X^T=H=HH^T=H^TH$$

Now a neat property of idepotent matrices is that all of their eigenvalues must be equal to zero or one. Letting $e$ denote a normalised eigenvector of $H$ with eigenvalue $l$, we can prove this as follows:

$$He=le\implies H(He)=H(le)$$ $$LHS=H^2e=He=le\;\;\; RHS=lHe=l^2e$$ $$\implies le=l^2e\implies l=0\text{ or }1$$

(note that $e$ cannot be zero as it must satisfy $e^Te=1$) Now because $H$ is idepotent, $I_n-H$ also is, because

$$(I_n-H)(I_n-H)=I-IH-HI+H^2=I_n-H$$

We also have the property that the sum of the eigenvalues equals the trace of the matrix, and $$tr(I_n-H)=tr(I_n)-tr(H)=n-tr(X(X^TX)^{-1}X^T)=n-tr((X^TX)^{-1}X^TX)$$ $$=n-tr(I_{p+1})=n-p-1$$

Hence $I-H$ must have $n-p-1$ eigenvalues equal to $1$ and $p+1$ eigenvalues equal to $0$.

Now we can use the spectral decomposition of $I-H=ADA^T$ where $D=\begin{pmatrix}I_{n-p-1} & 0_{[n-p-1]\times[p+1]}\\0_{[p+1]\times [n-p-1]} & 0_{[p+1]\times [p+1]}\end{pmatrix}$ and $A$ is orthogonal (because $I-H$ is symmetric) . A further property which is useful is that $HX=X$. This helps narrow down the $A$ matrix

$$HX=X\implies(I-H)X=0\implies ADA^TX=0\implies DA^TX=0$$ $$\implies (A^TX)_{ij}=0\;\;\;i=1,\dots,n-p-1\;\;\; j=1,\dots,p+1$$

and we get:

$$\frac{\sum_{i=1}(Y-\hat{Y_i})^2}{\sigma^2}=\frac{Y^TADA^TY}{\sigma^2}=\frac{\sum_{i=1}^{n-p-1}(A^TY)_i^2}{\sigma^2}$$

Now, under the model we have $Y\sim N(X\beta,\sigma^2I)$ and using standard normal theory we have $A^TY\sim N(A^TX\beta,\sigma^2A^TA)\sim N(A^TX\beta,\sigma^2I)$ showing that the components of $A^TY$ are independent. Now using the useful result, we have that $(A^TY)_i\sim N(0,\sigma^2)$ for $i=1,\dots,n-p-1$. The chi-square distribution with $n-p-1$ degrees of freedom for the sum of squared errors follows immediately.

• Wow, Thank you very much. It really is magnificent! Matrix form really pays off! In summary, SSE/$\sigma^2 = Y^T(I-H)Y$ and $I-H$ is idempotent. Idempotent matrices have eigenvalues either 0 or 1. So sum of eigenvalues is the number of eigenvalue 1. and $tr(I_n-H)= tr(I_n)-tr(H)=tr(I_n)-tr(X(X^T X)^-1 X^T)=tr(I_n)-tr((X^T X)^-1 X^T X)$ since $tr(AB)=tr(BA)$, and $tr(I_n-H)$ becomes n-p+1. and sum of eigenvalues of a matrix is sum of traces of the matrix! and $I-H$ can be expressed as $ADA^T$. So the first $Y^T(I-H)Y$ becomes $Y^TADA^TY$ with D with only n-p-1 diagonal 1's. – KH Kim Mar 21 '12 at 13:42
• Great answer!! Just to present another approach, we can instead choose to define a transformed multivariate normal variable $v := A'Y$ and it will still follow the same distribution $\mathcal{N}\left(0, \sigma^{2}I\right)$ if we use the affine property. Then the last fraction $\frac{Y'ADA'Y}{\sigma^{2}} = \frac{v'Dv}{\sigma^{2}} = \frac{v'\begin{bmatrix} I & 0\\0 & 0\end{bmatrix}v}{\sigma^{2}}= \sum_{i=1}^{\operatorname{tr}D} \left(\frac{v_{i}}{\sigma}\right)^{2}$. – Daeyoung Lim Feb 25 '16 at 15:13