Proof/Derivation of Residual Sum of Squares (Based on Introduction to Statistical Learning)

Question

On page 19 of the textbook Introduction to Statistical Learning (by James, Witten, Hastie and Tibshirani--it is freely downloadable on the web, and very good), the following is stated:

Consider a given estimate $$\hat{Y} = \hat{f}(x)$$ Assume for a moment that both $$\hat{f}, X$$ are fixed. Then, it is easy to show that:

$$\mathrm{E}(Y - \hat{Y})^2 = \mathrm{E}[f(X) + \epsilon - \hat{f}(X)]^2$$ $$ = [f(X) - \hat{f}(X)]^2 + \mathrm{Var}(\epsilon)$$

It is further explained that the first term represents the reducible error, and the second term represents the irreducible error.

I am not fully understanding how the authors arrive at this answer. I worked through the calculations as follows:

$$\mathrm{E}(Y - \hat{Y})^2 = \mathrm{E}[f(X) + \epsilon - \hat{f}(X)]^2$$

This simplifies to $[f(X) - \hat{f}(X) + \mathrm{E}[\epsilon]]^2 = [f(X) - \hat{f}(X)]^2$ assuming that $\mathrm{E}[\epsilon] = 0$. Where is the $\mathrm{Var}(x)$ indicated in the text coming from?

Any suggestions would be greatly appreciated.

Answer 1

Simply expand the square ...

$$[f(X)- \hat{f}(X) + \epsilon ]^2=[f(X)- \hat{f}(X)]^2 +2 [f(X)- \hat{f}(X)]\epsilon+ \epsilon^2$$

... and use linearity of expectations:

$$\mathrm{E}[f(X)- \hat{f}(X) + \epsilon ]^2=E[f(X)- \hat{f}(X)]^2 +2 E[(f(X)- \hat{f}(X))\epsilon]+ E[\epsilon^2]$$

Can you do it from there? (What things remain to be shown?)

Hint in response to comments: Show $E(\epsilon^2)=\text{Var}(\epsilon)$

Answer 2

\begin{equation} \ E[(Y−\hat{Y})^2] = E[(f(X)+\epsilon-\hat{f}(X))^2] = E[(f(X)-\hat{f}(X))^2 + \epsilon^2 + 2\epsilon(f(X)-\hat{f}(X))] = E[(f(X)-\hat{f}(X))^2] + E[\epsilon^2] + E[2\epsilon(f(X)-\hat{f}(X))] = E[(f(X)-\hat{f}(X))^2] + E[\epsilon^2] + 2(f(X)-\hat{f}(X))*E[\epsilon].......(1)\\ \end{equation} The Last term is zero as the expected value of irreducible error is zero. And lets see where variance come from. In general: \begin{equation} \ Var(X) = E[(X−\bar{X})^2] = E[X^2 - 2X\bar{X} + \bar{X}^2] = E[X^2] - E[2X\bar{X}] + E[\bar{X}^2]\\ \end{equation} The mean of X is a constant and so is the square of the mean of X. Therefore equation becomes, \begin{equation} \ Var(X) = E[X^2] - 2\bar{X}*E[X] + \bar{X}^2 = E[X^2] - 2\bar{X}*\bar{X} + \bar{X}^2 = E[X^2] - 2\bar{X}^2 + \bar{X}^2 = E[X^2] - \bar{X}^2\\ Hence,\\Var(\epsilon) = E[\epsilon^2] - \bar{\epsilon}^2\\ \end{equation} But mean of $\epsilon$ is zero. So, \begin{equation} \\Var(\epsilon) = E[\epsilon^2].....(2) \\ \end{equation} Now taking equation 1, whose last term is zero & equation 2: \begin{equation} \ E[(Y−\hat{Y})^2] = E[(f(X)-\hat{f}(X))^2] + E[\epsilon^2] = E[(f(X)-\hat{f}(X))^2] + Var(\epsilon) \end{equation}