Prove that $\frac{(n-2)s^2}{\sigma^2}\sim \chi^{2}_{n-2}$

Question

Consider the following simple linear regression model involving the $\epsilon_i$ error term,

$$y_i = \alpha + \beta x_i + \epsilon_i$$

such that,

$$\epsilon_i \sim \mathcal N(0,\sigma^2)$$

we know that if $Z_1, Z_2, ..., Z_n$ are independent, standard normal random variables,

$$Z_i \sim \mathcal N(0,1)$$

then the sum of their squares is distributed according to the chi-square distribution with $n$ degrees of freedom,

$$\sum _{i=1}^{n} Z_{i}^{2} \sim \chi^{2}_{n}$$

Now let's prove that $\frac{(n-2)s^2}{\sigma^2}\sim \chi^{2}_{n-2}$ such that,

$$s^2 = \frac{\sum _{i=1}^{n} \hat{\epsilon}_i^2}{n-2}$$

Proof:

$\require{cancel}$

$$\epsilon_i \sim \mathcal N(0,\sigma^2)$$

$$\frac{\epsilon_i - \cancelto{0}{\overline{\epsilon}}}{\sigma} \sim \mathcal N(0, 1)$$

$$\sum _{i=1}^{n} \left(\frac{\epsilon_i}{\sigma}\right)^2 \sim \chi^{2}_{n}$$

$$\sum _{i=1}^{n} \frac{\epsilon_i^2}{\sigma^2} \sim \chi^{2}_{n}$$

How do I now move from this expression $\sum _{i=1}^{n} \frac{\epsilon_i^2}{\sigma^2}$ which uses the error term $\epsilon_i$ to this expression $\sum _{i=1}^{n} \frac{\hat{\epsilon}_i^2}{\sigma^2}$ which uses the residual term $\hat{\epsilon}_i$?

Note:

I came across this answer but I got lost in the second part of the demonstration and I wanted to know if there is a simpler method that would lead to the same answer.

Answer 1

Without using the orthogonal change of variables in the linked answer, you can work under the general matrix setup of multiple linear regression. Here we are concerned with ordinary least squares.

The key result to be used here is the Fisher-Cochran theorem on distribution of quadratic forms (e.g. see page 185-186, 2nd edition of Linear Statistical Inference and Its Applications by Rao).

Suppose the model is $$y_i=\beta_0+\beta_1x_i+\varepsilon_i$$

Define the vectors $y=(y_1,\ldots,y_n)$, $\beta=(\beta_0,\beta_1)$ and $\varepsilon=(\varepsilon_1,\ldots,\varepsilon_n)$.

So in matrix form you have $$y=X\beta+\varepsilon\,,$$ where $X_{n\times k}$ with rank $k$ (here $k=2$) is the matrix of covariates (fixed) and $\varepsilon\sim N(0,\sigma^2 I_n)$.

By standard algebra of least squares, $$\hat y=X\hat\beta=X(X'X)^{-1}X'y=Hy\,,$$ where $H=X(X'X)^{-1}X'$ is the hat matrix. Note that $H$ is symmetric and idempotent.

The residual vector is

\begin{align} e=y-\hat y&=(I_n-H)y \\&=(I_n-H)(X\beta+\varepsilon) \\&=(I_n-H)\varepsilon \end{align}

Now $I_n-H$ is symmetric and idempotent because $H$ is so.

So the residual sum of squares is

\begin{align} e'e&=y'(I_n-H)'(I_n-H)y \\&=y'(I_n-H)y \\&=\varepsilon'(I_n-H)\varepsilon \end{align}

Therefore $$\frac{(n-2)s^2}{\sigma^2}=\frac{e'e}{\sigma^2}=\left(\frac{\varepsilon}{\sigma}\right)'(I_n-H)\frac{\varepsilon}{\sigma}$$

By a corollary of the Fisher-Cochran theorem, the above has a chi-square distribution because $(I_n-H)$ is idempotent, the degrees of freedom being $\operatorname{rank}(I_n-H)$.

As rank of an idempotent matrix equals its trace, $\operatorname{rank}(I_n-H)=\operatorname{tr}(I_n-H)=n-\operatorname{tr}(H)$.

And by properties of trace, $\operatorname{tr}(H)=\operatorname{tr}(X(X'X)^{-1}X')=\operatorname{tr}(X'X(X'X)^{-1})=\operatorname{tr}(I_k)=k$