This section of Introduction to Statistical Learning in R (page 19 in v6, statement 2.3) is motivating the difference between reducible and irreducible error (that is noted by $\epsilon$ and has mean zero).

Consider a given estimate $\hat{f}$ and a set of predictors $X$, which yields the prediction $\hat{Y} = \hat{f}(X)$. Assume for a moment that both $\hat{f}$ and $\hat{X}$ are fixed. Then, it is easy to show that

$E(Y - \hat{Y})^2 = E[f(X) + \epsilon - \hat{f}(X)]^2$

$= [f(X) - \hat{f}(X)]^2 + Var(\epsilon)$

I'm having trouble getting the intermediate steps here. I understand that the expectation of the error in prediction should involve the variance of $\epsilon$ but I'd like to understand the proof.

I expanded to

$E(Y - \hat{Y})^2 = E[f(X) - \hat{f}(X)]^2 + 2E(\epsilon)E[f(X) - \hat{f}(X)] + E(\epsilon)^2$

and I see that I have $E(\epsilon)$ and $E(\epsilon)^2$ terms that could lead to $Var(\epsilon)$, but I'm stuck trying to fit some of the basic expected value and variance manipulations to it.

  • $\begingroup$ Please add the [self-study] tag & read its wiki. Then tell us what you understand thus far, what you've tried & where you're stuck. We'll provide hints to help you get unstuck. $\endgroup$ Jan 17 '16 at 19:59
  • $\begingroup$ Because you tell us the mean of $\epsilon$ is $0$, you may substitute $0$ for $E(\epsilon)$ and $\operatorname{Var}(\epsilon)$ for $E(\epsilon^2)$ in your formula. What does that produce? $\endgroup$
    – whuber
    Jan 17 '16 at 20:24
  • $\begingroup$ OK, with that I solved it. Is it preferred for me to answer my own question with this information, or just edit my question with the answer? $\endgroup$ Jan 17 '16 at 20:38
  • 2
    $\begingroup$ The preference is to post an answer, please. $\endgroup$
    – whuber
    Jan 17 '16 at 21:07

\begin{align*} \mathbb{E}\left[(Y-\hat Y)^2\right] &=\mathbb{E}\left[\left(f(X)+\epsilon-\hat{f}(X)\right)^2\right] \\ &=\mathbb{E}\left[\left(f(X)+\epsilon-\hat{f}(X)\right) \left(f(X)+\epsilon-\hat{f}(X)\right)\right] \\ &=\mathbb{E}\left[\left(f(X)-\hat{f}(X)\right) \left(f(X)+\epsilon-\hat{f}(X)\right) +\epsilon \left(f(X)+\epsilon-\hat{f}(X)\right)\right] \\ &=\mathbb{E}\left[\left(f(X)-\hat{f}(X)\right)^2 +\epsilon \left(f(X)-\hat{f}(X)\right) +\epsilon \left(f(X)-\hat{f}(X)\right) +\epsilon^2\right] \\ \text{Because the expectation is linear}&\\ &=\mathbb{E}\left[\left(f(X)-\hat{f}(X)\right)^2\right] +\mathbb{E}\left[\epsilon^2\right] +2\mathbb{E}\left[\epsilon \left(f(X)-\hat{f}(X)\right)\right] \\ \text{Because the expectation of $f$ and $\hat{f}$ are constant}&\\ &=\left[f(X)-\hat{f}(X)\right]^2 +\mathbb{E}\left[\epsilon^2\right] +2\mathbb{E}\left[\epsilon \left(f(X)-\hat{f}(X)\right)\right] \\ \text{Because the mean of $\epsilon$ is zero}&\\ &=\left[f(X)-\hat{f}(X)\right]^2 +\mathbb{E}\left[\epsilon^2\right] \\ \text{Because the variance of $\epsilon$ is $\mathbb{E}(\epsilon^2)$}&\\ &=\left[f(X)-\hat{f}(X)\right]^2 + \text{Var}(\epsilon) \end{align*}

  • 1
    $\begingroup$ With self-study questions, we generally prefer giving hints rather a full answer. However, since the question is rather old, and the question hasn't been asked as part of a homework sheet that other students might copy later (but rather, a difficulty following a textbook), I think the provision of a full answer in this case is actually helpful. $\endgroup$
    – Silverfish
    Mar 8 '16 at 17:54
  • $\begingroup$ Ah, I see. The answer was actually self-study myself. I came looking for the answer to the same question, having prepared everything but the last two steps. I guess I wanted to get credit for my work :D $\endgroup$ Mar 8 '16 at 17:56
  • $\begingroup$ You can read our policy on self-study questions here, but I thought this was fine :) $\endgroup$
    – Silverfish
    Mar 8 '16 at 19:05
  • $\begingroup$ I appreciate the posted answer because I realized I hadn't properly solved it when I was going to post the solution to my own question. $\endgroup$ Mar 9 '16 at 21:23
  • $\begingroup$ why are $f$ and $\hat{f}$ constant? $\endgroup$
    – baxx
    Nov 9 '19 at 22:43

Not the answer you're looking for? Browse other questions tagged or ask your own question.