Variance term in bias-variance decomposition of linear regression

Question

In 'The Elements of Statistical Learning', the expression for bias-variance decomposition of linear-model is given as $$Err(x_0)=\sigma_\epsilon^2+E[f(x_0)-E\hat f(x_0)]^2+||h(x_0)||^2\sigma_\epsilon^2,$$ where $f(x_0)$ is the actual target function, $ \sigma_\epsilon^2$ is variance of random error in the model $y=f(x)+\epsilon$ and $\hat f(x)$ is the linear estimator of $f(x)$.

The variance term is troubling me here because the equation implies that the variance would be zero if the targets are noiseless, that is, $\sigma_\epsilon^2=0.$ But it does not make sense to me because even with zero noise I can still get different estimators $\hat f(x_0)$ for different training sets which implies variance is non-zero.

For example, suppose the target function $f(x_0)$ is a quadratic and the training data contains two points sampled at random from this quadratic; clearly, I will get a different linear fit everytime I sample two points randomly from the quadratic-target. Then how can variance be zero?

Can anyone help me find out what is wrong in my understanding of bias-variance decomposition?

Answer 1

There's always a lurking subtlety in treatments of bias and variance, and it's important to pay careful attention to it when studying. If you re-read the first few words of ESL in a section from that chapter, the authors to pay it some respect.

Discussions of error rate estimation can be confusing, because we have to make clear which quantities are fixed and which are random

The subtlety is what is fixed, and what is random.

In traditional treatments of linear regression, the data $X$ is treated as fixed and known. If you follow the arguments in ESL, you will find that the authors are also making this assumption. Under these assumptions, your example does not come into play, as the only remaining source of randomness in from the conditional distribution of $y$ given $X$. If it helps, you may want to replace the notation $Err(x_0)$ in your mind with $Err(x_0 \mid X)$.

That is not to say that your concern is invalid, it is certainly true that the selection of training data does indeed introduce randomness in our model algorithm, and a diligent practitioner will attempt to quantify the effect of this randomness on their outcomes. In fact, you can see quite clearly that the common practices of bootstrapping and cross-validation explicitly incorporate these sources of randomness into their inferences.

To derive an explicit mathematical expression for the bias and variance of a linear model in the context of a random training data set, one would need to make some assumptions about the structure of the randomness in the $X$ data. This would involve some suppositions on the distribution of $X$. This can be done, but has not become part of the mainstream expositions of these ideas.