0
$\begingroup$

I'm having trouble understanding something from the linear regression chapter of Elements of Statistical Learning.

We have a fixed $N\times p$ matrix $\mathbf{X}$ ($N$ inputs with $p$ predictors) that gives us the observations $\mathbf{y}$ which are "uncorrelated and have constant variance $\sigma^2$." Our predictions $\mathbf{\hat{y}}$ are given using the standard least squares solution, $\mathbf{\hat{y}}=\mathbf{X}\hat{\beta}=\mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$. Apparently, an unbiased estimator of the variance $\sigma^2$ is given by:

$$ \hat{\sigma}^2 = \frac{1}{N-p-1}\sum_{i=1}^N(y_i - \hat{y_i})^2 $$ Or the residual sum of squares divided by the degrees of freedom. This leads to: $$ (N-p-1)\hat{\sigma}^2 \sim \sigma^2\chi^2_{N-p-1} $$

I'm familiar with Bessel's Correction, but I still couldn't reconcile these two. Proof that the first one is an unbiased estimator or that the similarity in the second one is true would be greatly appreciated!

Improve this question
$\endgroup$
4
  • $\begingroup$ On the first thing, what's wrong with the first link under "Related" in the sidebar? This would have been offered to you as a possible answer when you posted. Note that the $p$ in the answer is one larger than yours (it includes the constant in its count of $p$). As for the second, what about this $\endgroup$ – Glen_b May 8 '17 at 7:34
  • $\begingroup$ Strange, I didn't see that one suggested when I was writing the post or searching but it's definitely the right answer -- maybe because I was always searching for "N - p - 1". $\endgroup$ – Walker Harrison May 8 '17 at 13:32
  • $\begingroup$ Oh, okay, sorry about that. $\endgroup$ – Glen_b May 8 '17 at 14:45
  • $\begingroup$ I just went through part of this on a different question and I think this is quite confusing in the book: They claim that the expression for $\hat{\sigma}^2$ is an unbiased estimator of $\sigma^2$ before they impose the assumption that $Y = X\beta + \epsilon$. I cannot make this work. It seems to me that this is not correct, i.e. you cannot prove $\mathbb{E}(\hat{\sigma}^2) = \sigma^2$ unless you already know that $\mathbb{E}(\textbf{y} | X) = \mathbb{E}(\hat{\textbf{y}} | X)$, and this is not actually an assumption or a known fact at this point. $\endgroup$ – T_M Jun 3 '20 at 10:36