# Linear regression: Unbiased estimator of the variance of outputs [duplicate]

I'm having trouble understanding something from the linear regression chapter of Elements of Statistical Learning.

We have a fixed $N\times p$ matrix $\mathbf{X}$ ($N$ inputs with $p$ predictors) that gives us the observations $\mathbf{y}$ which are "uncorrelated and have constant variance $\sigma^2$." Our predictions $\mathbf{\hat{y}}$ are given using the standard least squares solution, $\mathbf{\hat{y}}=\mathbf{X}\hat{\beta}=\mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T\mathbf{y}$. Apparently, an unbiased estimator of the variance $\sigma^2$ is given by:

$$\hat{\sigma}^2 = \frac{1}{N-p-1}\sum_{i=1}^N(y_i - \hat{y_i})^2$$ Or the residual sum of squares divided by the degrees of freedom. This leads to: $$(N-p-1)\hat{\sigma}^2 \sim \sigma^2\chi^2_{N-p-1}$$

I'm familiar with Bessel's Correction, but I still couldn't reconcile these two. Proof that the first one is an unbiased estimator or that the similarity in the second one is true would be greatly appreciated!

• On the first thing, what's wrong with the first link under "Related" in the sidebar? This would have been offered to you as a possible answer when you posted. Note that the $p$ in the answer is one larger than yours (it includes the constant in its count of $p$). As for the second, what about this May 8, 2017 at 7:34
• Strange, I didn't see that one suggested when I was writing the post or searching but it's definitely the right answer -- maybe because I was always searching for "N - p - 1". May 8, 2017 at 13:32
• Oh, okay, sorry about that. May 8, 2017 at 14:45
• I just went through part of this on a different question and I think this is quite confusing in the book: They claim that the expression for $\hat{\sigma}^2$ is an unbiased estimator of $\sigma^2$ before they impose the assumption that $Y = X\beta + \epsilon$. I cannot make this work. It seems to me that this is not correct, i.e. you cannot prove $\mathbb{E}(\hat{\sigma}^2) = \sigma^2$ unless you already know that $\mathbb{E}(\textbf{y} | X) = \mathbb{E}(\hat{\textbf{y}} | X)$, and this is not actually an assumption or a known fact at this point.
– T_M
Jun 3, 2020 at 10:36