If we take $\mu$ to be the true regression function, and we estimate $\mu$ by $\hat\mu$ from the available data, which is random, and therefore so is the estimate, which we may denote by $\hat M_n$ to signify that we used a finite $n$ for the estimation.
I am a little confused about the following argument which attempts to show the prediction error of the overall 'method': $$ MSE(\hat M_n(x)) = E( (Y-\hat M_n(x))^2|X)=\dots=\sigma^2(x) + (\mu(x)-E(\hat M_n(x)))^2 + Var(\hat M_n(x)) $$
This is the bias-variance decomposition, and as I understand it:
- 1st term: represents the difficulty of predicting $Y$ at $X=x$, it is the unpredictable fluctuation that is present even with the best prediction
- 2nd term: The bias in using $\hat M_n$ to approximate $\mu$, that is, this is our approximation error
- 3rd term: variance in our estimate of the regression function
What I am finding difficult to picture is the bias variance trade-off, why exactly does there have to be one and what does it imply about our choice of $\hat M_n $?