Hello I have two questions. I am trying to understand this formulas. But I cant understand how the MS Residual is different from the common aleatory error

enter image description here

Another thing, I just labeled each value with the with their supposed corresponding formula, it is correct?

  • 2
    $\begingroup$ What is "common aleatory error"? $\endgroup$
    – Zhanxiong
    Feb 8 at 19:58
  • 1
    $\begingroup$ $MS_{residual}$ probably means "Mean Squared Residual" and doesn't refer to any specific error (hence the "approx" in the last line.) $\endgroup$
    – jbowman
    Feb 8 at 20:04

1 Answer 1


Since the OLS model assumptions are that $y_i=\beta^\top x_i + \epsilon_i$ with $\epsilon_i\stackrel{\text{i.i.d.}}{\sim}\mathcal N(0,\sigma^2)$, $\text{MS}_{\text{residual}}:=\frac 1 n \sum_i(y_i-\hat{\beta}^\top x_i)^2$ is (only) an approximation of the variance $\sigma^2$ for two reasons

1. $\hat \beta\approx\beta\in\mathbb{R}^d$ (holds in expectation)

The OLS estimator $\hat\beta$ is unbiased as $\mathbb{E}\left[\hat \beta\right]=\beta$ and the expected error converges to 0 in the limit since $\mathbb{E}\left[||\hat\beta-\beta||^2\right]=\frac {\sigma^2d}{n}$

2. $\frac 1 n \sum_i(y_i-\beta^\top x_i)^2=\frac 1 n \sum_i\epsilon_i^2 \approx\sigma^2$ (holds in limit)

By the law of large numbers, as $n \rightarrow \infty$ we have $\frac 1 n\sum_i\epsilon_i^2\rightarrow\mathbb{E}\left[\epsilon_i^2\right]=\sigma^2$

Search "Statistical guarantees for ordinary least squares" for proofs and such

  • $\begingroup$ Nice answer! It would be great if you could specify a bit better "holds in expectation" and "holds in limit" as well as adding more complete infos about your reference to that paper (I guess it's a paper). Thank you $\endgroup$
    – jmarkov
    Feb 9 at 5:11
  • $\begingroup$ These are very standard undergrad level results, as such I do not think you need/would be able to find a paper, but maybe some free course material. You could easily prove them yourself or find them online. But alright, I will include them here directly. $\endgroup$ Feb 9 at 9:00
  • 1
    $\begingroup$ Someone asking that kind of question likely is not familiar with terms like "holds in expectation". Also, the way you reported Statistical guarantees for ordinary least square it sounded like the title of an article or book. I think it just needed to be easier to follow for a beginner. $\endgroup$
    – jmarkov
    Feb 10 at 1:13
  • $\begingroup$ Thank you for your time ! $\endgroup$
    – RodParedes
    Feb 10 at 16:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.