# Proof for Irreducible Error statement in ISLR page 19 [duplicate]

This section of Introduction to Statistical Learning in R (page 19 in v6, statement 2.3) is motivating the difference between reducible and irreducible error (that is noted by $$\epsilon$$ and has mean zero).

Consider a given estimate $$\hat{f}$$ and a set of predictors $$X$$, which yields the prediction $$\hat{Y} = \hat{f}(X)$$. Assume for a moment that both $$\hat{f}$$ and $$\hat{X}$$ are fixed. Then, it is easy to show that

$$E(Y - \hat{Y})^2 = E[f(X) + \epsilon - \hat{f}(X)]^2$$

$$= [f(X) - \hat{f}(X)]^2 + Var(\epsilon)$$

I'm having trouble getting the intermediate steps here. I understand that the expectation of the error in prediction should involve the variance of $$\epsilon$$ but I'd like to understand the proof.

I expanded to

$$E(Y - \hat{Y})^2 = E[f(X) - \hat{f}(X)]^2 + 2E(\epsilon)E[f(X) - \hat{f}(X)] + E(\epsilon)^2$$

and I see that I have $$E(\epsilon)$$ and $$E(\epsilon)^2$$ terms that could lead to $$Var(\epsilon)$$, but I'm stuck trying to fit some of the basic expected value and variance manipulations to it.

• Please add the [self-study] tag & read its wiki. Then tell us what you understand thus far, what you've tried & where you're stuck. We'll provide hints to help you get unstuck. Jan 17 '16 at 19:59
• Because you tell us the mean of $\epsilon$ is $0$, you may substitute $0$ for $E(\epsilon)$ and $\operatorname{Var}(\epsilon)$ for $E(\epsilon^2)$ in your formula. What does that produce?
– whuber
Jan 17 '16 at 20:24
• OK, with that I solved it. Is it preferred for me to answer my own question with this information, or just edit my question with the answer? Jan 17 '16 at 20:38
– whuber
Jan 17 '16 at 21:07

\begin{align*} \mathbb{E}\left[(Y-\hat Y)^2\right] &=\mathbb{E}\left[\left(f(X)+\epsilon-\hat{f}(X)\right)^2\right] \\ &=\mathbb{E}\left[\left(f(X)+\epsilon-\hat{f}(X)\right) \left(f(X)+\epsilon-\hat{f}(X)\right)\right] \\ &=\mathbb{E}\left[\left(f(X)-\hat{f}(X)\right) \left(f(X)+\epsilon-\hat{f}(X)\right) +\epsilon \left(f(X)+\epsilon-\hat{f}(X)\right)\right] \\ &=\mathbb{E}\left[\left(f(X)-\hat{f}(X)\right)^2 +\epsilon \left(f(X)-\hat{f}(X)\right) +\epsilon \left(f(X)-\hat{f}(X)\right) +\epsilon^2\right] \\ \text{Because the expectation is linear}&\\ &=\mathbb{E}\left[\left(f(X)-\hat{f}(X)\right)^2\right] +\mathbb{E}\left[\epsilon^2\right] +2\mathbb{E}\left[\epsilon \left(f(X)-\hat{f}(X)\right)\right] \\ \text{Because the expectation of f and \hat{f} are constant}&\\ &=\left[f(X)-\hat{f}(X)\right]^2 +\mathbb{E}\left[\epsilon^2\right] +2\mathbb{E}\left[\epsilon \left(f(X)-\hat{f}(X)\right)\right] \\ \text{Because the mean of \epsilon is zero}&\\ &=\left[f(X)-\hat{f}(X)\right]^2 +\mathbb{E}\left[\epsilon^2\right] \\ \text{Because the variance of \epsilon is \mathbb{E}(\epsilon^2)}&\\ &=\left[f(X)-\hat{f}(X)\right]^2 + \text{Var}(\epsilon) \end{align*}
• why are $f$ and $\hat{f}$ constant?