# Understanding bias-variance tradeoff derivation

I am reading the chapter of bias-variance tradeoff of The elements of statistical learning and I have doubt in the formula at page 29. Let the data arise from a model such that $$Y = f(x)+\epsilon$$ where $\epsilon$ is random number with expected value $\hat{\epsilon} = E[\epsilon]=0$ and Variance $E[(\epsilon - \hat\epsilon)^2]=E[\epsilon^2]=\sigma^2$. Let the expected value of error of the model is $$E[(Y-f_k(x))^2]$$ where $f_k(x)$ is the prediction of $x$ of our learner. According to the book, the error is $$E[(Y-f_k(x))^2]=\sigma^2+Bias(f_k)^2+Var(f_k(x)).$$

My question is why bias term is not 0? developing the formula of the error I see $$E[(Y-f_k(x))^2]=\\ E[(f(x)+\epsilon-f_k(x))^2]=\\ E[(f(x)-f_k(x))^2]+2E[(f(x)-f_k(x))\epsilon]+E[\epsilon^2]=\\ Var(f_k(x))+2E[(f(x)-f_k(x))\epsilon]+\sigma^2$$

as $\epsilon$ is an independent random number $2E[(f(x)-f_k(x))\epsilon]=2E[(f(x)-f_k(x))]E[\epsilon]=0$

Where I am wrong?

-

You are not wrong, but you made an error in one step since $E[(f(x)-f_k(x))^2] \ne Var(f_k(x))$. $E[(f(x)-f_k(x))^2]$ is $\text{MSE}(f_k(x)) = Var(f_k(x)) + \text{Bias}^2(f_k(x))$.

\begin{align*} E[(Y-f_k(x))^2]& = E[(f(x)+\epsilon-f_k(x))^2] \\ &= E[(f(x)-f_k(x))^2]+2E[(f(x)-f_k(x))\epsilon]+E[\epsilon^2]\\ &= E\left[\left(f(x) - E(f_k(x)) + E(f_k(x))-f_k(x) \right)^2 \right] + 2E[(f(x)-f_k(x))\epsilon]+\sigma^2 \\ & = Var(f_k(x)) + \text{Bias}^2(f_k(x)) + \sigma^2. \end{align*}

Note: $E[(f_k(x)-E(f_k(x)))(f(x)-E(f_k(x))] = E[f_k(x)-E(f_k(x))](f(x)-E(f_k(x))) = 0.$

-
In case of binary outcomes, Is there an equivalent proof with cross entropy as error measure? – emanuele Mar 28 at 15:22
It doesn't work out quite so well with a binary response. See Ex 7.2 in the second edition of "The Elements of Statistical Learning". – Matthew Drury Mar 28 at 21:25