Understanding the bias variance trade-off of the regression function

Question

If we take $\mu$ to be the true regression function, and we estimate $\mu$ by $\hat\mu$ from the available data, which is random, and therefore so is the estimate, which we may denote by $\hat M_n$ to signify that we used a finite $n$ for the estimation.

I am a little confused about the following argument which attempts to show the prediction error of the overall 'method': $$ MSE(\hat M_n(x)) = E( (Y-\hat M_n(x))^2|X)=\dots=\sigma^2(x) + (\mu(x)-E(\hat M_n(x)))^2 + Var(\hat M_n(x)) $$

This is the bias-variance decomposition, and as I understand it:

• 1st term: represents the difficulty of predicting $Y$ at $X=x$, it is the unpredictable fluctuation that is present even with the best prediction
• 2nd term: The bias in using $\hat M_n$ to approximate $\mu$, that is, this is our approximation error
• 3rd term: variance in our estimate of the regression function

What I am finding difficult to picture is the bias variance trade-off, why exactly does there have to be one and what does it imply about our choice of $\hat M_n $?

Answer 1

See section 2.2.2 of ISL or section 7.3 of ElemStatLearn. The trade-off applies when selecting the best model among models of different complexity or flexibility. Generally, as the complexity/flexibility increases, the variance (3rd term) increases and the squared bias (2nd term) decreases. Note that the variance and squared bias are both always positive, so $MSE > \sigma^2$ for all models. The trade-off refers to finding the "sweet spot" (minimum MSE) where the model has low variance and low bias.