Mean Squared Error as Reducible and Irreducible Component

I am having problem with a basic proof . I want to decompose Mean Square Error into Reducible and Irreducible parts as shown below, but I cannot go from the step 2 to step 3.

\begin{align} \mathbb E(Y - \hat Y)^2 &=\mathbb E\left[f(X) + \epsilon - \hat f(X)\right]^2\\&= \mathbb E\left[\left(f(X) - \hat f(X)\right)^2 + 2\epsilon\left(f(X) - \hat f(X)\right)+\epsilon^2\right]\\ &=\left(f(X) - \hat f(X)\right)^2 + \mathrm{Var}(\epsilon) \end{align}

• Please add the [self-study] tag & read its wiki. Then tell us what you understand thus far, what you've tried & where you're stuck. We'll provide hints to help you get unstuck. Mar 17 '16 at 11:46
• While I agree with @gung that this question should be tagged as self-study, I think it is quite clear where the OP has managed to do the work and where they have come stuck, which is why I have voted to leave the question open. (I think I am inclined to be more generous because this is a standard proof, rather than a textbook or course question.) Mar 17 '16 at 11:53
• @Silverfish, I also voted to leave open. This should have the tag & follow our policy, but not all SS Qs need to be closed right away. Mar 17 '16 at 11:58

Recall that the expectation of $Y$ is conditional on $x$, so every $E(\cdot)$ below is actually $E(\cdot|x)$. Thus all terms with only $x$ are constants inside the expectation.

Also, remember that $E(\epsilon) = 0$ and thus $E(\epsilon^2) = Var(\epsilon)$.

\begin{align*} E(Y - \hat{Y})^2 & = E\left[(f(x) - \hat{f}(x))^2 + 2\epsilon (f(x) - \hat{f}(x)) + \epsilon^2 \right]\\ & = E\left[(f(x) - \hat{f}(x))^2 \right] + 2E(\epsilon) E(f(x) - \hat{f}(x)) + E(\epsilon^2)\\ & = (f(x) - \hat{f}(x))^2 + Var(\epsilon). \end{align*}

• You're saying that given $X = x$, we are actually stating $E(Y - \hat{Y} | X = x) = (f(x) - \hat{f}(x))^2 + Var(\epsilon)$? Apr 22 '20 at 1:43
• Also, what's an intuitive way to understand the relationship of $Y|X$? For reference, the original ISLR (2017) pp. 19 states $Y = f(X) + \epsilon$, where $X$ is a set of predictors and $\epsilon$ is a random error term w/ mean 0. Why is $Y$ not a RV? Apr 22 '20 at 1:53

Adding to @Greenparker, the first term of the last line is decomposed as below

$=(f(x)-\hat{f(x)}+E(\hat{f(x)})-E(\hat{f(x)})^2+Var(\epsilon)$

$=(\underbrace{E(\hat{f(x)})-f(x)}_{Bias})^2+(\underbrace{E(\hat{f(x)})-\hat{f(x)})^2}_{Var(\hat{f(x))}}+Var(\epsilon)$