Understanding bias-variance tradeoff derivation I am reading the chapter on the bias-variance tradeoff in The elements of statistical learning and I don't understand the formula on page 29. Let the data arise from a model such that $$ Y = f(x)+\varepsilon$$ where $\varepsilon$ is random number with expected value $\hat{\varepsilon} = E[\epsilon]=0$ and Variance $E[(\varepsilon - \hat\varepsilon)^2]=E[\varepsilon^2]=\sigma^2$. Let the expected value of error of the model be
$$
E[(Y-f_k(x))^2]
$$
where $f_k(x)$ is the prediction of $x$ of our learner $k$. According to the book, the error is
$$
\newcommand{\Bias}{\rm Bias} \newcommand{\Var}{\rm Var}
E[(Y-f_k(x))^2]=\sigma^2+\Bias(f_k)^2+\Var(f_k(x)).
$$
My question is: Why is the bias term not $0$? Developing the formula for the error I see:
\begin{align}
E[(Y-f_k(x))^2] &= \\
E[(f(x)+\varepsilon-f_k(x))^2] &= \\[8pt]
E[(f(x)-f_k(x))^2] + \\ 
2E[(f(x)-f_k(x))\varepsilon] + E[\varepsilon^2] &= \Var(f_k(x))+2E[(f(x)-f_k(x))\epsilon]+\sigma^2
\end{align}
as $\varepsilon$ is an independent random number $2E[(f(x)-f_k(x))\varepsilon] = 2E[(f(x)-f_k(x))]E[\varepsilon]=0$.
Where I am wrong?
 A: A few more steps of the Bias - Variance decomposition
Indeed, the full derivation is rarely given in textbooks as it involves a lot of uninspiring algebra. Here is a more complete derivation using notation from the book "Elements of Statistical Learning" on page 223

If we assume that $Y = f(X) + \epsilon$ and $E[\epsilon] = 0$ and $Var(\epsilon) = \sigma^2_\epsilon$ then we can derive the expression for the expected prediction error of a regression fit $\hat f(X)$ at an input $X = x_0$ using squared error loss
$$Err(x_0) = E[ (Y - \hat f(x_0) )^2 | X = x_0]$$
For notational simplicity let $\hat f(x_0) = \hat f$, $f(x_0) = f$ and recall that $E[f] = f$ and $E[Y] = f$
\begin{aligned}
 E[ (Y - \hat f)^2 ] &= E[(Y - f + f - \hat f )^2]
 \\
 & = E[(y - f)^2] + E[(f - \hat f)^2] + 2 E[(f - \hat f)(y - f)]
 \\
 & =  E[(f + \epsilon - f)^2] + E[(f - \hat f)^2]  + 2E[fY - f^2 - \hat f Y + \hat f f]
 \\
 & = E[\epsilon^2] + E[(f - \hat f)^2] + 2( f^2 - f^2 - f E[\hat f] + f E[\hat f] )
 \\
 & = \sigma^2_\epsilon + E[(f - \hat f)^2] + 0
\end{aligned}
For the term $E[(f - \hat f)^2]$ we can use a similar trick as above, adding and subtracting $E[\hat f]$ to get
\begin{aligned}
 E[(f - \hat f)^2] & = E[(f + E[\hat f] - E[\hat f] - \hat f)^2] 
 \\
 & = E \left[ f - E[\hat f] \right]^2 + E\left[ \hat f - E[ \hat f] \right]^2 
 \\
 & = \left[ f - E[\hat f] \right]^2 + E\left[ \hat f - E[ \hat f] \right]^2 
 \\
 & = Bias^2[\hat f] + Var[\hat f] 
\end{aligned}
Putting it together
$$E[ (Y - \hat f)^2 ]  =  \sigma^2_\epsilon + Bias^2[\hat f] + Var[\hat f] $$

Some comments on why $E[\hat f Y] = f E[\hat f]$
Taken from Alecos Papadopoulos here
Recall that $\hat f$ is the predictor we have constructed based on the $m$ data points $\{(x^{(1)},y^{(1)}),...,(x^{(m)},y^{(m)}) \}$ so we can write $\hat f = \hat f_m$ to remember that. 
On the other hand $Y$ is the prediction we are making on a new data point $(x^{(m+1)},y^{(m+1)})$ by using the model constructed on the $m$ data points above. So the Mean Squared Error can be written as 
$$ E[\hat f_m(x^{(m+1)}) - y^{(m+1)}]^2$$
Expanding the equation from the previous section 
$$E[\hat f_m Y]=E[\hat f_m (f+ \epsilon)]=E[\hat f_m f+\hat f_m \epsilon]=E[\hat f_m f]+E[\hat f_m \epsilon]$$
The last part of the equation can be viewed as
$$ E[\hat f_m(x^{(m+1)}) \cdot \epsilon^{(m+1)}] = 0$$
Since we make the following assumptions about the point $x^{(m+1)}$: 


*

*It was not used when constructing $\hat f_m$ 

*It is independent of all other observations $\{(x^{(1)},y^{(1)}),...,(x^{(m)},y^{(m)}) \}$

*It is independent of $\epsilon^{(m+1)}$
Other sources with full derivations


*

*https://en.wikipedia.org/wiki/Bias%E2%80%93variance_tradeoff#Derivation

*https://robjhyndman.com/files/2-biasvardecomp.pdf

*http://www.inf.ed.ac.uk/teaching/courses/mlsc/Notes/Lecture4/BiasVariance.pdf
A: You are not wrong, but you made an error in one step since $E[(f(x)-f_k(x))^2] \ne Var(f_k(x))$. $E[(f(x)-f_k(x))^2]$ is $\text{MSE}(f_k(x)) = Var(f_k(x)) + \text{Bias}^2(f_k(x))$.
\begin{align*}
E[(Y-f_k(x))^2]& = E[(f(x)+\epsilon-f_k(x))^2] \\
&= E[(f(x)-f_k(x))^2]+2E[(f(x)-f_k(x))\epsilon]+E[\epsilon^2]\\
&= E\left[\left(f(x) - E(f_k(x)) + E(f_k(x))-f_k(x) \right)^2 \right] + 2E[(f(x)-f_k(x))\epsilon]+\sigma^2 \\
& = Var(f_k(x)) + \text{Bias}^2(f_k(x)) + \sigma^2.
\end{align*}
Note: $E[(f_k(x)-E(f_k(x)))(f(x)-E(f_k(x))] = E[f_k(x)-E(f_k(x))](f(x)-E(f_k(x))) = 0.$
