In 'The Elements of Statistical Learning', the expression for bias-variance decomposition of linear-model is given as $$Err(x_0)=\sigma_\epsilon^2+E[f(x_0)-E\hat f(x_0)]^2+||h(x_0)||^2\sigma_\epsilon^2,$$ where $f(x_0)$ is the actual target function, $ \sigma_\epsilon^2$ is variance of random error in the model $y=f(x)+\epsilon$ and $\hat f(x)$ is the linear estimator of $f(x)$.

The variance term is troubling me here because the equation implies that the variance would be zero if the targets are noiseless, that is, $\sigma_\epsilon^2=0.$ But it does not make sense to me because even with zero noise I can still get different estimators $\hat f(x_0)$ for different training sets which implies variance is non-zero.

For example, suppose the target function $f(x_0)$ is a quadratic and the training data contains two points sampled at random from this quadratic; clearly, I will get a different linear fit everytime I sample two points randomly from the quadratic-target. Then how can variance be zero?

Can anyone help me find out what is wrong in my understanding of bias-variance decomposition?


1 Answer 1


There's always a lurking subtlety in treatments of bias and variance, and it's important to pay careful attention to it when studying. If you re-read the first few words of ESL in a section from that chapter, the authors to pay it some respect.

Discussions of error rate estimation can be confusing, because we have to make clear which quantities are fixed and which are random

The subtlety is what is fixed, and what is random.

In traditional treatments of linear regression, the data $X$ is treated as fixed and known. If you follow the arguments in ESL, you will find that the authors are also making this assumption. Under these assumptions, your example does not come into play, as the only remaining source of randomness in from the conditional distribution of $y$ given $X$. If it helps, you may want to replace the notation $Err(x_0)$ in your mind with $Err(x_0 \mid X)$.

That is not to say that your concern is invalid, it is certainly true that the selection of training data does indeed introduce randomness in our model algorithm, and a diligent practitioner will attempt to quantify the effect of this randomness on their outcomes. In fact, you can see quite clearly that the common practices of bootstrapping and cross-validation explicitly incorporate these sources of randomness into their inferences.

To derive an explicit mathematical expression for the bias and variance of a linear model in the context of a random training data set, one would need to make some assumptions about the structure of the randomness in the $X$ data. This would involve some suppositions on the distribution of $X$. This can be done, but has not become part of the mainstream expositions of these ideas.

  • $\begingroup$ Thanks a lot for clearing out the fact that authors have assumed $X$ to be fixed, so the expectation here is w.r.t. $Y|X$ not $(X,Y)$. But we can write $E=E_XE_{Y|X}$, which means that treating X as random we will get $Var(\hat f(x_0))=E_X[||h(x_0)||^2\sigma_\epsilon^2]$. It would be still be zero if $\sigma_\epsilon^2$ is zero. I had a similar doubt about this equation, you can find out my derivation at this post: stats.stackexchange.com/questions/307110/… $\endgroup$ Oct 16, 2017 at 16:21
  • $\begingroup$ My guess there is that the authors are assuming the model is correctly specified, i.e. includes all and only the relevant predictors with the correct transformations. I'd have to go back to the book instead of relying on my memory to confirm though. $\endgroup$ Oct 16, 2017 at 16:34
  • $\begingroup$ If by 'correctly specified' you mean that the target function is indeed linear then I understand that zero noise would imply zero bias. But it turns out even if the target function is not linear, we get the exact same expression for the variance. $\endgroup$ Oct 16, 2017 at 16:39
  • 1
    $\begingroup$ It's true, but in that case "correctly specified" would mean that you were using linear regression to fit a model including the correct predictors. So if the true relationship is quadratic, then you would be assuming your model includes the quadratic terms. $\endgroup$ Oct 16, 2017 at 17:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.