I am currently reading An Introduction to Statistical Learning by James, Witten, Hastie, and Tibshirani, and I am stuck on one of the leaps they take when defining reducible and irreducible error. They state that for a given set of input and output variable $X$ and $Y$, respectively, there is a function $f$ which we estimate, defined as $\hat f$. Using $\hat f$ we then obtain the our predictions of $Y$ which is called $\hat{Y}$ such that $\hat Y = \hat f(X)$. On page 19, equation 2.3, they give the following equation:

$$E(Y - \hat Y)^2 = E[f(X) + \epsilon - \hat f(X)]^2$$ $$ = [f(X) - \hat f(X)]^2 + Var(\epsilon)$$

They then go on to say that $[f(X) - \hat f(X)]^2$ is the reducible error while $Var(\epsilon)$ is the irreducible error.

I am by no means a mathematician and have tried some derivations but cannot reach that conclusion on my own, particularly in deriving $Var(\epsilon)$. Thank you.


Because $E\left[\textrm{constant}\right]=\textrm{constant}$, (and $f$ and $\hat f$ are constants), $E\left[\epsilon\right]=0$, and $\textrm{Var}(\epsilon) = E\left[\left(\epsilon - E\left[\epsilon\right]\right)^2\right]$. You get the expected results. \begin{align} E(Y - \hat Y)^2 &= E\left[f(X) + \epsilon - \hat f(X)\right]^2\\ &= E\left[\left(f(X) - \hat f(X)\right)+\epsilon\right]^2\\ &=E\left[\left(f(X) - \hat f(X)\right)^2+2\epsilon\left(f(X) - \hat f(X)\right)+\epsilon^2\right]\\ &=E\left[\left(f(X) - \hat f(X)\right)^2\right]+E\left[2\epsilon\left(f(X) - \hat f(X)\right)\right]+E\left[\epsilon^2\right]\\ &=\left(f(X) - \hat f(X)\right)^2+0+E\left[\epsilon^2\right] \end{align}

  • $\begingroup$ what is the sample space here? Why do you consider only e to be a random variable? Shouldn't X also be considered an r.v. as we are trying to compute the average squared error between prediction and actual value of Y for all Xs in our population? What is our sample space if both X and e are random? Can you please expand your answer to address those points? $\endgroup$ Dec 8 '19 at 3:11
  • $\begingroup$ Moreover, as this book (I have the seventh printing) explicitly says in the introduction (where notation is introduced) that random variables are denoted with capital letters. So it would seem that X is a random variable and, therefore, so is f(X) and f^(X). As such the E[...] cannot be removed. Also, by the same notation, e should be E. Am I missing something? $\endgroup$ Dec 8 '19 at 4:04
  • $\begingroup$ @Gilles Just a question about the notation. The expected value is a function so I would think the the parentheses would encapsulate the arguments for this function. It seems like E(Y -Yhat)^2 would be the square of the expected difference of Y and Yhat. E[(Y-Yhat)^2] would be the expected square difference between Y and Yhat. Are they using some other notation that I'm not familiar with? $\endgroup$ Jul 9 '20 at 17:43

Not the answer you're looking for? Browse other questions tagged or ask your own question.