0
$\begingroup$

I'm having trouble understanding something from the linear regression chapter of Elements of Statistical Learning.

We have a fixed $N\times p$ matrix $\mathbf{X}$ ($N$ inputs with $p$ predictors) that gives us the observations $\mathbf{y}$ which are "uncorrelated and have constant variance $\sigma^2$." Our predictions $\mathbf{\hat{y}}$ are given using the standard least squares solution, $\mathbf{\hat{y}}=\mathbf{X}\hat{\beta}=\mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$. Apparently, an unbiased estimator of the variance $\sigma^2$ is given by:

$$ \hat{\sigma}^2 = \frac{1}{N-p-1}\sum_{i=1}^N(y_i - \hat{y_i})^2 $$ Or the residual sum of squares divided by the degrees of freedom. This leads to: $$ (N-p-1)\hat{\sigma}^2 \sim \sigma^2\chi^2_{N-p-1} $$

I'm familiar with Bessel's Correction, but I still couldn't reconcile these two. Proof that the first one is an unbiased estimator or that the similarity in the second one is true would be greatly appreciated!

| cite | improve this question | |
$\endgroup$
  • $\begingroup$ On the first thing, what's wrong with the first link under "Related" in the sidebar? This would have been offered to you as a possible answer when you posted. Note that the $p$ in the answer is one larger than yours (it includes the constant in its count of $p$). As for the second, what about this $\endgroup$ – Glen_b May 8 '17 at 7:34
  • $\begingroup$ Strange, I didn't see that one suggested when I was writing the post or searching but it's definitely the right answer -- maybe because I was always searching for "N - p - 1". $\endgroup$ – Walker Harrison May 8 '17 at 13:32
  • $\begingroup$ Oh, okay, sorry about that. $\endgroup$ – Glen_b May 8 '17 at 14:45
  • $\begingroup$ I just went through part of this on a different question and I think this is quite confusing in the book: They claim that the expression for $\hat{\sigma}^2$ is an unbiased estimator of $\sigma^2$ before they impose the assumption that $Y = X\beta + \epsilon$. I cannot make this work. It seems to me that this is not correct, i.e. you cannot prove $\mathbb{E}(\hat{\sigma}^2) = \sigma^2$ unless you already know that $\mathbb{E}(\textbf{y} | X) = \mathbb{E}(\hat{\textbf{y}} | X)$, and this is not actually an assumption or a known fact at this point. $\endgroup$ – T_M Jun 3 at 10:36