10
$\begingroup$

On page 19 of the textbook Introduction to Statistical Learning (by James, Witten, Hastie and Tibshirani--it is freely downloadable on the web, and very good), the following is stated:

Consider a given estimate $$\hat{Y} = \hat{f}(x)$$ Assume for a moment that both $$\hat{f}, X$$ are fixed. Then, it is easy to show that:

$$\mathrm{E}(Y - \hat{Y})^2 = \mathrm{E}[f(X) + \epsilon - \hat{f}(X)]^2$$ $$ = [f(X) - \hat{f}(X)]^2 + \mathrm{Var}(\epsilon)$$

It is further explained that the first term represents the reducible error, and the second term represents the irreducible error.

I am not fully understanding how the authors arrive at this answer. I worked through the calculations as follows:

$$\mathrm{E}(Y - \hat{Y})^2 = \mathrm{E}[f(X) + \epsilon - \hat{f}(X)]^2$$

This simplifies to $[f(X) - \hat{f}(X) + \mathrm{E}[\epsilon]]^2 = [f(X) - \hat{f}(X)]^2$ assuming that $\mathrm{E}[\epsilon] = 0$. Where is the $\mathrm{Var}(x)$ indicated in the text coming from?

Any suggestions would be greatly appreciated.

$\endgroup$
6
  • 1
    $\begingroup$ Because this is from a textbook, you should add the self-study tag to your question. See stats.stackexchange.com/tags/self-study/info $\endgroup$ Jul 31 '14 at 20:05
  • 2
    $\begingroup$ Your notation is mystifying because $\mathrm{E}(Y - \hat{Y})^2 = \mathrm{E}[f(X) + \epsilon - \hat{f}(X)]^2$ literally means the square of the expectation. Assuming $\mathrm{E}(\epsilon)=0$, this immediately reduces to $(f(X)-\hat{f}(X)+\mathrm{E}(\epsilon))^2$ = $(f(X)-\hat{f}(X))^2$. Evidently, then, what you really want to compute is the expectation of the square, $\mathrm{E}[(f(X)-\hat{f}(X)+\epsilon)^2]$. But if so, the very first step in your derivation makes no sense. Could you edit the question to clear this up? $\endgroup$
    – whuber
    Jul 31 '14 at 20:30
  • $\begingroup$ Hmm.. I see what you mean. I didn't see that simplification at first (i.e. that $E[f(X)+\epsilon - \hat{f}(X)]^2 = [f(X) - \hat{f}(X) + E(\epsilon)]^2 = [f(X) - \hat{f}(X)]^2$. But that further adds to my confusion about how we get $[f(X) - \hat{f}(X)]^2 + Var(\epsilon)$ as the answer. Where is the Var(\epsilon) coming from? I will edit the question to reflect this clarification. $\endgroup$
    – wellington
    Jul 31 '14 at 20:42
  • $\begingroup$ I was not pointing to a simplification, but to a distinction: the expectation of the square does not equal the square of the expectation. Even after the edits your question does not seem to recognize this crucial fact. $\endgroup$
    – whuber
    Aug 1 '14 at 1:01
  • 1
    $\begingroup$ The issue that I was having was the notation in the book. The way I was initially thinking of the problem, I was approaching it as $\mathrm{E}[(Y - \hat{Y})^2] = (\mathrm{E}[f(X) + \epsilon - \hat{f}(X)])^2$ i.e. quantity squared. What I later learned was, the book was trying to imply that $\mathrm{E}[f(X) + \epsilon - \hat{f}(X)]^2$ actually means $\mathrm{E}([f(X) + \epsilon - \hat{f}(X)]^2)$ I personally think this notation is a bit confusing, but it's how it's written in the text. I agree that it's important to remember that $\mathrm{E}[X^2] \neq \mathrm{E}[X]^2$ $\endgroup$
    – wellington
    Aug 1 '14 at 1:08
6
$\begingroup$

Simply expand the square ...

$$[f(X)- \hat{f}(X) + \epsilon ]^2=[f(X)- \hat{f}(X)]^2 +2 [f(X)- \hat{f}(X)]\epsilon+ \epsilon^2$$

... and use linearity of expectations:

$$\mathrm{E}[f(X)- \hat{f}(X) + \epsilon ]^2=E[f(X)- \hat{f}(X)]^2 +2 E[(f(X)- \hat{f}(X))\epsilon]+ E[\epsilon^2]$$

Can you do it from there? (What things remain to be shown?)

Hint in response to comments: Show $E(\epsilon^2)=\text{Var}(\epsilon)$

$\endgroup$
10
  • $\begingroup$ I actually was able to get that far in the time I've been trying at this problem since. One of the confusions that I had the first time around was that I was treating the entire term, $\mathrm{E}[...]$ to be squared, rather than just squaring the inside, i.e. $\mathrm{E}([...]^2)$. I understand why $\mathrm{E}[f(X) - \hat{f}(X)]^2$ becomes $[f(X) - \hat{f}(X)]^2$ since it is just a number, and the expected value of a real number is just the number. What I don't understand is how $2\mathrm{E}[f(X)-\hat{f}(X))\epsilon] + \mathrm{E}[\epsilon^2]$ becomes $\mathrm{Var}(\epsilon)$... $\endgroup$
    – wellington
    Aug 1 '14 at 0:16
  • $\begingroup$ see my additional hint. What now remains to be shown? $\endgroup$
    – Glen_b
    Aug 1 '14 at 0:30
  • $\begingroup$ Well we know that $\mathrm{E}(\epsilon^2) = \mathrm{Var}(\epsilon) + \mathrm{E}[\epsilon]^2$. The only thing I can think of is that we now apply the assumption that $\mathrm{E}[\epsilon] = 0$, therefore $(\mathrm{E}[\epsilon])^2 = 0$. Am I on the right track? $\endgroup$
    – wellington
    Aug 1 '14 at 0:35
  • $\begingroup$ Yes, that's it. So what's left? And what's assumed about those quantities? $\endgroup$
    – Glen_b
    Aug 1 '14 at 0:35
  • 1
    $\begingroup$ @George see the conditions in the question which tell us we're at a fixed value of $X$. $\endgroup$
    – Glen_b
    Apr 11 '16 at 17:03
0
$\begingroup$

\begin{equation} \ E[(Y−\hat{Y})^2] = E[(f(X)+\epsilon-\hat{f}(X))^2] = E[(f(X)-\hat{f}(X))^2 + \epsilon^2 + 2\epsilon(f(X)-\hat{f}(X))] = E[(f(X)-\hat{f}(X))^2] + E[\epsilon^2] + E[2\epsilon(f(X)-\hat{f}(X))] = E[(f(X)-\hat{f}(X))^2] + E[\epsilon^2] + 2(f(X)-\hat{f}(X))*E[\epsilon].......(1)\\ \end{equation} The Last term is zero as the expected value of irreducible error is zero. And lets see where variance come from. In general: \begin{equation} \ Var(X) = E[(X−\bar{X})^2] = E[X^2 - 2X\bar{X} + \bar{X}^2] = E[X^2] - E[2X\bar{X}] + E[\bar{X}^2]\\ \end{equation} The mean of X is a constant and so is the square of the mean of X. Therefore equation becomes, \begin{equation} \ Var(X) = E[X^2] - 2\bar{X}*E[X] + \bar{X}^2 = E[X^2] - 2\bar{X}*\bar{X} + \bar{X}^2 = E[X^2] - 2\bar{X}^2 + \bar{X}^2 = E[X^2] - \bar{X}^2\\ Hence,\\Var(\epsilon) = E[\epsilon^2] - \bar{\epsilon}^2\\ \end{equation} But mean of $\epsilon$ is zero. So, \begin{equation} \\Var(\epsilon) = E[\epsilon^2].....(2) \\ \end{equation} Now taking equation 1, whose last term is zero & equation 2: \begin{equation} \ E[(Y−\hat{Y})^2] = E[(f(X)-\hat{f}(X))^2] + E[\epsilon^2] = E[(f(X)-\hat{f}(X))^2] + Var(\epsilon) \end{equation}

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.