7
$\begingroup$

This section of Introduction to Statistical Learning in R (page 19 in v6, statement 2.3) is motivating the difference between reducible and irreducible error (that is noted by $\epsilon$ and has mean zero).

Consider a given estimate $\hat{f}$ and a set of predictors $X$, which yields the prediction $\hat{Y} = \hat{f}(X)$. Assume for a moment that both $\hat{f}$ and $\hat{X}$ are fixed. Then, it is easy to show that

$E(Y - \hat{Y})^2 = E[f(X) + \epsilon - \hat{f}(X)]^2$

$= [f(X) - \hat{f}(X)]^2 + Var(\epsilon)$

I'm having trouble getting the intermediate steps here. I understand that the expectation of the error in prediction should involve the variance of $\epsilon$ but I'd like to understand the proof.

I expanded to

$E(Y - \hat{Y})^2 = E[f(X) - \hat{f}(X)]^2 + 2E(\epsilon)E[f(X) - \hat{f}(X)] + E(\epsilon)^2$

and I see that I have $E(\epsilon)$ and $E(\epsilon)^2$ terms that could lead to $Var(\epsilon)$, but I'm stuck trying to fit some of the basic expected value and variance manipulations to it.

$\endgroup$
4
  • $\begingroup$ Please add the [self-study] tag & read its wiki. Then tell us what you understand thus far, what you've tried & where you're stuck. We'll provide hints to help you get unstuck. $\endgroup$ Commented Jan 17, 2016 at 19:59
  • $\begingroup$ Because you tell us the mean of $\epsilon$ is $0$, you may substitute $0$ for $E(\epsilon)$ and $\operatorname{Var}(\epsilon)$ for $E(\epsilon^2)$ in your formula. What does that produce? $\endgroup$
    – whuber
    Commented Jan 17, 2016 at 20:24
  • $\begingroup$ OK, with that I solved it. Is it preferred for me to answer my own question with this information, or just edit my question with the answer? $\endgroup$ Commented Jan 17, 2016 at 20:38
  • 2
    $\begingroup$ The preference is to post an answer, please. $\endgroup$
    – whuber
    Commented Jan 17, 2016 at 21:07

1 Answer 1

15
$\begingroup$

\begin{align*} \mathbb{E}\left[(Y-\hat Y)^2\right] &=\mathbb{E}\left[\left(f(X)+\epsilon-\hat{f}(X)\right)^2\right] \\ &=\mathbb{E}\left[\left(f(X)+\epsilon-\hat{f}(X)\right) \left(f(X)+\epsilon-\hat{f}(X)\right)\right] \\ &=\mathbb{E}\left[\left(f(X)-\hat{f}(X)\right) \left(f(X)+\epsilon-\hat{f}(X)\right) +\epsilon \left(f(X)+\epsilon-\hat{f}(X)\right)\right] \\ &=\mathbb{E}\left[\left(f(X)-\hat{f}(X)\right)^2 +\epsilon \left(f(X)-\hat{f}(X)\right) +\epsilon \left(f(X)-\hat{f}(X)\right) +\epsilon^2\right] \\ \text{Because the expectation is linear}&\\ &=\mathbb{E}\left[\left(f(X)-\hat{f}(X)\right)^2\right] +\mathbb{E}\left[\epsilon^2\right] +2\mathbb{E}\left[\epsilon \left(f(X)-\hat{f}(X)\right)\right] \\ \text{Because the expectation of $f$ and $\hat{f}$ are constant}&\\ &=\left[f(X)-\hat{f}(X)\right]^2 +\mathbb{E}\left[\epsilon^2\right] +2\mathbb{E}\left[\epsilon \left(f(X)-\hat{f}(X)\right)\right] \\ \text{Because the mean of $\epsilon$ is zero}&\\ &=\left[f(X)-\hat{f}(X)\right]^2 +\mathbb{E}\left[\epsilon^2\right] \\ \text{Because the variance of $\epsilon$ is $\mathbb{E}(\epsilon^2)$}&\\ &=\left[f(X)-\hat{f}(X)\right]^2 + \text{Var}(\epsilon) \end{align*}

$\endgroup$
8
  • 1
    $\begingroup$ With self-study questions, we generally prefer giving hints rather a full answer. However, since the question is rather old, and the question hasn't been asked as part of a homework sheet that other students might copy later (but rather, a difficulty following a textbook), I think the provision of a full answer in this case is actually helpful. $\endgroup$
    – Silverfish
    Commented Mar 8, 2016 at 17:54
  • 2
    $\begingroup$ why are $f$ and $\hat{f}$ constant? $\endgroup$
    – baxx
    Commented Nov 9, 2019 at 22:43
  • 2
    $\begingroup$ Second @baxx question. Why are $f$ and $\hat{f}$ constant? $\endgroup$
    – ngram
    Commented Nov 6, 2020 at 23:21
  • 1
    $\begingroup$ Because this is relative to a single fit. $\endgroup$ Commented Nov 7, 2020 at 1:27
  • 1
    $\begingroup$ Why f and f^ are constant?: Suppose you have a 1 X 3 matrix. You have X (input variable), Y (real measured value of outcome) and a Y hat (you used a linear function f hat, or in layman terms a formula to get Y hat). Now Y = f(X) + error. Here, whenever you try to measure Y, it will always differ from the previous measurement, but the underlying f(X) remains the same. It would always be the ever-changing error term making the difference. And when you calculate Y hat as an estimation for Y, you would use the same f hat on the same X, resulting in the same Y hat. So you see the nuance here. $\endgroup$ Commented Mar 28, 2021 at 15:48

Not the answer you're looking for? Browse other questions tagged or ask your own question.