1
$\begingroup$

An excerpt about Simple Linear Regression and RSE from Introduction to Statistical Learning

Can someone help me understand this? I've been trying to wrap my head around but the more I do, I end up being confused.

It says RSE is a measure of Standard deviation of non removable error term epsilon. But in the equation, RSE just calculates distance between prediction and the actual value.

So, my guess is the y value in the dataset is more or less off by some value. Maybe due to the fact that it wasn't recorded correctly or maybe due to some other factor. To adjust this, we're introducing the epsilon term.

But, then again, how come RSE is a SD of epsilon?

$\endgroup$

2 Answers 2

1
$\begingroup$

As the book states, "an error term is associated with each observation". The irreducible error. If you achieve a 0 RSE, then great but this is likely never going to happen. The reason is many: poor sampling of target, poor sampling of predictors, not enough data vs model complexity (see Bias-Variance trade-off), poor estimation of the true relationship between predictors and targets, time or spatial varying distribution model for predictors or targets, leaving out necessary predictors, quantization in sampling or models parameters, etc.

$\endgroup$
0
$\begingroup$

Let's say that your model is

$$ y = f(X) + \epsilon $$

where $\epsilon$ is the noise. Then $\hat y = f(X)$ are the predictions and $e = y - \hat y$ are the residuals. Mean squared error estimates

$$ \operatorname{MSE}(y, \hat y) = E[(y - \hat y)^2] = E[e^2] $$

The variance of the errors could be estimated from the residuals

$$\begin{align} \operatorname{Var}(\epsilon) &= \operatorname{Var}(e) \\ &= E[(e - E[e])^2] \\ &= E[(y - \hat y - E[e])^2] \end{align}$$

If your estimator is unbiased, then $E[y] = E[\hat y]$, hence $E[y - \hat y] = E[y] - E[\hat y] = 0$, so the above formula reduces to

$$ E[(y - \hat y - 0)^2] = E[e^2] $$

So if you have additive noise and an unbiased estimator, the variance of the errors is the same as the mean squared error. If your estimator is biased, then the mean of the residuals is not zero, so an additional bias term enters the equation, and it is not error variance anymore. Also if the noise is not additive (but e.g. multiplicative) then the variance of the residuals calculated like $y - \hat y$ will not tell you about the variance of $\epsilon$.

$\endgroup$
2
  • $\begingroup$ I almost understand your answer but there's one clarification I need about Expectation of Variance formula. You said: Var(ϵ )= Var(e) = E[(e−E[e])^2]. But shouldn't it be E(e^2) - E(e)^2 $\endgroup$ Dec 7, 2022 at 5:36
  • $\begingroup$ @TANAYJOSHI that's the same thing en.wikipedia.org/wiki/Variance#Definition $\endgroup$
    – Tim
    Dec 7, 2022 at 10:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.