5
$\begingroup$

I'm reading Introduction to Statistical Learning. The relevant part is referenced here: Proof/Derivation of Residual Sum of Squares (Based on Introduction to Statistical Learning).

When the author shows graphs that illustrate "Bias vs Variance Tradeoff" (as in Figure 2.12), the ${\rm Var}(\varepsilon)$ is always $1$ (note the dashed lines in the figures):

three graphs of MSE on y-axis; flexibility on x-axis, and Var(epsilon) is always at 1

The conditions of $\varepsilon$ are clarified elsewhere, as on page 16:

$\varepsilon$ is a random error term, which is independent of $X$ and has mean zero.

... and there is some explanation about going from "random error term" to "irreducible error":

However, even if it were possible to form a perfect estimate for $f$, so that our estimated response took the form $\hat{Y} = f(X)$, our prediction would still have some error in it! This is because $Y$ is also a function of $\varepsilon$, which, by definition, cannot be predicted using $X$. Therefore, variability associated with $\varepsilon$ also affects the accuracy of our predictions.

But I don't see anywhere in the other SO questions, nor in the book: why is $Var(\varepsilon)$ always at 1?

  • Is it because the "mean is zero"? I don't think so; I could describe a dataset with mean of zero but a variance of $\ne 1$.
  • Is it because, as described elsewhere, the "the error term $\varepsilon$ is normally distributed"? I don't know enough about the normal distribution; is the variance of a normal distribution is always equal to some value?

EDIT

In looking for help in Wikipedia's MSE article, I expected to find a consistent formula with the "three fundamental quantities" (i.e., the variance, the bias, and the variance of the error terms), but I didn't. Can someone tell me why the Wikipedia doesn't list the variance of error terms:

$$\operatorname{MSE}(\hat{\theta})=\operatorname{Var}(\hat{\theta})+ \left(\operatorname{Bias}(\hat{\theta},\theta)\right)^2$$

Improve this question
$\endgroup$
6
  • 2
    $\begingroup$ When doing examples, people will often just set the variance of the error term $\epsilon$ equal to 1. If they didn't want to fix a value for $Var(\epsilon) = \sigma^2$ then they wouldn't have been able to make those plots with particular numbers on the side. $\endgroup$
    – jld
    Aug 9, 2016 at 1:23
  • $\begingroup$ So is it just convention? Why 1? 1 is convenient? No other assumptions about error term can lead us to this effect? ( I realize I am probably over thinking something that is, by definition, unknowable...) $\endgroup$
    – Nate Anderson
    Aug 9, 2016 at 1:25
  • $\begingroup$ Author does say, "...the irreducible error will always provide an upper bound on the accuracy of our prediction for $Y$. This bound is almost always unknown in practice" Is this as good as admitting that 1 is an arbitrary choice? $\endgroup$
    – Nate Anderson
    Aug 9, 2016 at 1:27
  • 3
    $\begingroup$ also this has absolutely no bearing on real data. This is purely for the sake of the examples that they're doing. In real life $Var(\epsilon)$ could be anything $\endgroup$
    – jld
    Aug 9, 2016 at 1:29
  • $\begingroup$ Thanks @Chaconne, you should've answered so I could upvote a tortoise. Can we calculate $Var(\epsilon)$ in real life? Or is that the "unknown" author describes in practice $\endgroup$
    – Nate Anderson
    Aug 9, 2016 at 2:25

1 Answer 1

Reset to default
3
$\begingroup$

It isn't because the mean is $0$ or because the error term is normally distributed. In fact, the normal distribution is the only 'named' distribution where the mean and the variance are independent of each other (see: What is the most surprising characterization of the Gaussian (normal) distribution?).

More generally, my strong guess is that the purpose of setting the variance of the errors equal to $1$ is pedagogical. Everything in the figures can be related to the variance of the error term because the unit of measurement in the figures is $1$ and that was set as the variance of the error term.

Regarding the Wikipedia article, be aware that the variance of theta is a function of the variance of the error term, so ${\rm Var}(\hat\theta)$ does include ${\rm Var}(\varepsilon)$ (it's just out of sight).

Improve this answer
$\endgroup$
4
  • $\begingroup$ Thank you. This characterization of normal distributions - you're saying I could have both conditions: normal distribution and mean of 0, and still know absolutely nothing about variance? Relatedly, is it true that a mean of zero is a characteristic/definition of error terms while the variance of error terms could be whatever I want and they're still error terms? $\endgroup$
    – Nate Anderson
    Aug 9, 2016 at 2:42
  • 1
    $\begingroup$ @TheRedPea, yes. For a regression model, the mean of the errors is 0 by definition. But the variance can be any positive value. $\endgroup$
    – gung - Reinstate Monica
    Aug 9, 2016 at 3:53
  • $\begingroup$ The importance of the concept of "mean of errors approaching zero"... I did not realize it was so fundamental as described on NIST , but the way the describe it... Please confirm: It does not mean there is no "drift" , that would be wrong interpretation,, but that regression models do not attempt to account for such drift? ( as the regression which attempted to do that in, the case of drift, would have a very nonzero error?) Thanks again, gung, these dialogues are helpful for me... $\endgroup$
    – Nate Anderson
    Aug 9, 2016 at 6:47
  • 1
    $\begingroup$ @TheRedPea, that's really a different question (& I'm not sure I follow your question). You should ask this as a new question to get a better answer & so the information won't be buried in comments. $\endgroup$
    – gung - Reinstate Monica
    Aug 9, 2016 at 11:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.