I am having problem with a basic proof . I want to decompose Mean Square Error into Reducible and Irreducible parts as shown below, but I cannot go from the step 2 to step 3.

\begin{align} \mathbb E(Y - \hat Y)^2 &=\mathbb E\left[f(X) + \epsilon - \hat f(X)\right]^2\\&= \mathbb E\left[\left(f(X) - \hat f(X)\right)^2 + 2\epsilon\left(f(X) - \hat f(X)\right)+\epsilon^2\right]\\ &=\left(f(X) - \hat f(X)\right)^2 + \mathrm{Var}(\epsilon) \end{align}

  • 1
    $\begingroup$ Please add the [self-study] tag & read its wiki. Then tell us what you understand thus far, what you've tried & where you're stuck. We'll provide hints to help you get unstuck. $\endgroup$ Commented Mar 17, 2016 at 11:46
  • 2
    $\begingroup$ While I agree with @gung that this question should be tagged as self-study, I think it is quite clear where the OP has managed to do the work and where they have come stuck, which is why I have voted to leave the question open. (I think I am inclined to be more generous because this is a standard proof, rather than a textbook or course question.) $\endgroup$
    – Silverfish
    Commented Mar 17, 2016 at 11:53
  • $\begingroup$ @Silverfish, I also voted to leave open. This should have the tag & follow our policy, but not all SS Qs need to be closed right away. $\endgroup$ Commented Mar 17, 2016 at 11:58

2 Answers 2


Recall that the expectation of $Y$ is conditional on $x$, so every $E(\cdot)$ below is actually $E(\cdot|x)$. Thus all terms with only $x$ are constants inside the expectation.

Also, remember that $E(\epsilon) = 0$ and thus $E(\epsilon^2) = Var(\epsilon)$.

\begin{align*} E(Y - \hat{Y})^2 & = E\left[(f(x) - \hat{f}(x))^2 + 2\epsilon (f(x) - \hat{f}(x)) + \epsilon^2 \right]\\ & = E\left[(f(x) - \hat{f}(x))^2 \right] + 2E(\epsilon) E(f(x) - \hat{f}(x)) + E(\epsilon^2)\\ & = (f(x) - \hat{f}(x))^2 + Var(\epsilon). \end{align*}

  • $\begingroup$ You're saying that given $X = x$, we are actually stating $E(Y - \hat{Y} | X = x) = (f(x) - \hat{f}(x))^2 + Var(\epsilon)$? $\endgroup$
    – Zhulu
    Commented Apr 22, 2020 at 1:43
  • $\begingroup$ Also, what's an intuitive way to understand the relationship of $Y|X$? For reference, the original ISLR (2017) pp. 19 states $Y = f(X) + \epsilon$, where $X$ is a set of predictors and $\epsilon$ is a random error term w/ mean 0. Why is $Y$ not a RV? $\endgroup$
    – Zhulu
    Commented Apr 22, 2020 at 1:53

Adding to @Greenparker, the first term of the last line is decomposed as below




Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.