2
$\begingroup$

As per my notes, the key step in the proof that the sum of squares of residuals in regression is $\chi^{2}_{n-2}$ is the fact that $e_{i} = y_{i} - \hat{y}_{i}$ has a mean 0 and variance $\sigma^{2}$. This is the reason why $\dfrac{e_{i}}{\sigma}$ is standard normal and the sum of squares of the residuals turns out to be $\chi^{2}_{n-2}$.

The problem I have with this is that I derived the variance of a residual as $\sigma^{2}[1 - \frac{1}{n} - \frac{(x_{i} - \overline{x})^{2}}{S_{xx}}]$ when I was doing the predicted value of a new observation using the model.

Where is the logical flaw in the proof that the sum of squares of residuals is $\chi^{2}_{n-2}$ and how do we derive the fact that $\sum_{i=1}^{n}(\frac{e_i^{2}}{\sigma^2})$ is $\chi^{2}_{n-2}$?

$\endgroup$
  • $\begingroup$ Please add the [self-study] tag & read its wiki. $\endgroup$ – gung Jul 16 '16 at 1:53
  • $\begingroup$ Try to find the distribution of $e_i$. $\endgroup$ – Stat Jul 16 '16 at 15:45
1
$\begingroup$

Okay, so it turns out that the answer is not at all obvious as it was made out to be in the lecture videos I was watching (nice set of videos posted here ). We can solve it for the general case of multiple linear regression like so:

The model is $ \mathbf{Y} = \mathbf{X}\mathbf{\beta} + \mathbf{\epsilon}$. The fitted values are $ \hat{\mathbf{Y}} = \mathbf{X}\hat{\beta} = \mathbf{HY}$, where $\mathbf{H}$ is the hat matrix. From the above hat matrix notation, $\mathbf{e = Y - \hat{Y} = (I - H)Y = (I - H)(X\beta+\epsilon) = (I - H)\epsilon}$

The sum of the residual squares, $ SSE = \sum_{i=1}^{n} e_{i}^{2} = \mathbf{e^{'} e = \epsilon^{'}(I - H)\epsilon}$. This follows from the symmetric nature and the idempotence of $\mathbf{(I - H)}$.

So, $\dfrac{SSE}{\sigma^{2}} = \dfrac{\mathbf{\epsilon^{'}(I - H)\epsilon}}{\sigma^{2}} = \mathbf{\dfrac{\epsilon^{'}}{\sigma} (I - H) \dfrac{\epsilon}{\sigma}}$.

This final formula is now in the quadratic form $\mathbf{X = \dfrac{Y^{'}MY}{\sigma^{2}}}$, where $\mathbf{Y \sim N(0, \sigma^{2}I})$ and $\mathbf{M}$ is symmetric and idempotent. Using the properties of orthogonal factorization of M, we can show that the distribution of $\mathbf{X}$ is $\chi^{2}_{tr(M)}$, which completes this rather neat proof.

Much of the needed algebra was inspired by the answers to this question - In simple linear regression, where does the formula for the variance of the residuals come from?

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.