2
$\begingroup$

Hello I have two questions. I am trying to understand this formulas. But I cant understand how the MS Residual is different from the common aleatory error

enter image description here

Another thing, I just labeled each value with the with their supposed corresponding formula, it is correct?

$\endgroup$
2
  • 2
    $\begingroup$ What is "common aleatory error"? $\endgroup$
    – Zhanxiong
    Commented Feb 8, 2023 at 19:58
  • 1
    $\begingroup$ $MS_{residual}$ probably means "Mean Squared Residual" and doesn't refer to any specific error (hence the "approx" in the last line.) $\endgroup$
    – jbowman
    Commented Feb 8, 2023 at 20:04

1 Answer 1

2
$\begingroup$

Since the OLS model assumptions are that $y_i=\beta^\top x_i + \epsilon_i$ with $\epsilon_i\stackrel{\text{i.i.d.}}{\sim}\mathcal N(0,\sigma^2)$, $\text{MS}_{\text{residual}}:=\frac 1 n \sum_i(y_i-\hat{\beta}^\top x_i)^2$ is (only) an approximation of the variance $\sigma^2$ for two reasons

1. $\hat \beta\approx\beta\in\mathbb{R}^d$ (holds in expectation)

The OLS estimator $\hat\beta$ is unbiased as $\mathbb{E}\left[\hat \beta\right]=\beta$ and the expected error converges to 0 in the limit since $\mathbb{E}\left[||\hat\beta-\beta||^2\right]=\frac {\sigma^2d}{n}$

2. $\frac 1 n \sum_i(y_i-\beta^\top x_i)^2=\frac 1 n \sum_i\epsilon_i^2 \approx\sigma^2$ (holds in limit)

By the law of large numbers, as $n \rightarrow \infty$ we have $\frac 1 n\sum_i\epsilon_i^2\rightarrow\mathbb{E}\left[\epsilon_i^2\right]=\sigma^2$

Search "Statistical guarantees for ordinary least squares" for proofs and such

$\endgroup$
4
  • $\begingroup$ Nice answer! It would be great if you could specify a bit better "holds in expectation" and "holds in limit" as well as adding more complete infos about your reference to that paper (I guess it's a paper). Thank you $\endgroup$
    – jmarkov
    Commented Feb 9, 2023 at 5:11
  • $\begingroup$ These are very standard undergrad level results, as such I do not think you need/would be able to find a paper, but maybe some free course material. You could easily prove them yourself or find them online. But alright, I will include them here directly. $\endgroup$ Commented Feb 9, 2023 at 9:00
  • 1
    $\begingroup$ Someone asking that kind of question likely is not familiar with terms like "holds in expectation". Also, the way you reported Statistical guarantees for ordinary least square it sounded like the title of an article or book. I think it just needed to be easier to follow for a beginner. $\endgroup$
    – jmarkov
    Commented Feb 10, 2023 at 1:13
  • $\begingroup$ Thank you for your time ! $\endgroup$
    – RodParedes
    Commented Feb 10, 2023 at 16:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.