In lecture notes I am reading which are based on Tibshirani book on Statistical Learning in R, I am seeing that from the residuals $r_i = Y_i - x_{i}^{T}\hat{\beta}$ the usual estimate for $\sigma^2$ is: $$\sigma^2 = \frac{1}{n - p}\sum_{i = 1}^{n}r_i^2$$ Where $i$ is $i$-th sample, $n$ is number of samples and $p$ is number of predictor variables of $x_i$. If I understand correctly $n - p$ is number of degrees of freedom, but what I do not understand is how did we come to this conclusion. Is there any way to understand how did we come up to scaling by factor $\frac{1}{n - p}$. Note that I am only entering the statistics field, and that my previous experience is only on probability theory.

  • $\begingroup$ Have you searched our site? $\endgroup$ – whuber Feb 27 '20 at 23:01
  • $\begingroup$ Yes, I did, and I don't like the answers. Most of them are either showing that estimate of variance will be unbiased if we use this, or use "deegres of freedom" in abstract terms or a reference to a wikipedia page, which last time I watched, many wiki contributors have said to be corrupted with giberish. I want either some substantial mathematical proof of this or a good explanation. $\endgroup$ – GreatDuke Feb 28 '20 at 10:31
  • $\begingroup$ You do a disservice to Wikipedia: although a small number of articles are problematic, on basic statistical matters it consistently is more complete, accurate, and authoritative than almost all other sources you can find with Web searches. Sure, that's just my opinion, but it's based on extensive experience including ten years here on CV in which I have consulted Wikipedia on several thousand statistical topics and compared its articles to other Web resources I could find. I chose the duplicate threads in part because they do include substantial mathematical derivations and explanations. $\endgroup$ – whuber Feb 28 '20 at 14:13