If we take $\mu$ to be the true regression function, and we estimate $\mu$ by $\hat\mu$ from the available data, which is random, and therefore so is the estimate, which we may denote by $\hat M_n$ to signify that we used a finite $n$ for the estimation.

I am a little confused about the following argument which attempts to show the prediction error of the overall 'method': $$ MSE(\hat M_n(x)) = E( (Y-\hat M_n(x))^2|X)=\dots=\sigma^2(x) + (\mu(x)-E(\hat M_n(x)))^2 + Var(\hat M_n(x)) $$

This is the bias-variance decomposition, and as I understand it:

  • 1st term: represents the difficulty of predicting $Y$ at $X=x$, it is the unpredictable fluctuation that is present even with the best prediction
  • 2nd term: The bias in using $\hat M_n$ to approximate $\mu$, that is, this is our approximation error
  • 3rd term: variance in our estimate of the regression function

What I am finding difficult to picture is the bias variance trade-off, why exactly does there have to be one and what does it imply about our choice of $\hat M_n $?


See section 2.2.2 of ISL or section 7.3 of ElemStatLearn. The trade-off applies when selecting the best model among models of different complexity or flexibility. Generally, as the complexity/flexibility increases, the variance (3rd term) increases and the squared bias (2nd term) decreases. Note that the variance and squared bias are both always positive, so $MSE > \sigma^2$ for all models. The trade-off refers to finding the "sweet spot" (minimum MSE) where the model has low variance and low bias.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.