2
$\begingroup$

I am having problem with a basic proof . I want to decompose Mean Square Error into Reducible and Irreducible parts as shown below, but I cannot go from the step 2 to step 3.

\begin{align} \mathbb E(Y - \hat Y)^2 &=\mathbb E\left[f(X) + \epsilon - \hat f(X)\right]^2\\&= \mathbb E\left[\left(f(X) - \hat f(X)\right)^2 + 2\epsilon\left(f(X) - \hat f(X)\right)+\epsilon^2\right]\\ &=\left(f(X) - \hat f(X)\right)^2 + \mathrm{Var}(\epsilon) \end{align}

$\endgroup$
3
  • 1
    $\begingroup$ Please add the [self-study] tag & read its wiki. Then tell us what you understand thus far, what you've tried & where you're stuck. We'll provide hints to help you get unstuck. $\endgroup$ Mar 17 '16 at 11:46
  • 2
    $\begingroup$ While I agree with @gung that this question should be tagged as self-study, I think it is quite clear where the OP has managed to do the work and where they have come stuck, which is why I have voted to leave the question open. (I think I am inclined to be more generous because this is a standard proof, rather than a textbook or course question.) $\endgroup$
    – Silverfish
    Mar 17 '16 at 11:53
  • $\begingroup$ @Silverfish, I also voted to leave open. This should have the tag & follow our policy, but not all SS Qs need to be closed right away. $\endgroup$ Mar 17 '16 at 11:58
2
$\begingroup$

Recall that the expectation of $Y$ is conditional on $x$, so every $E(\cdot)$ below is actually $E(\cdot|x)$. Thus all terms with only $x$ are constants inside the expectation.

Also, remember that $E(\epsilon) = 0$ and thus $E(\epsilon^2) = Var(\epsilon)$.

\begin{align*} E(Y - \hat{Y})^2 & = E\left[(f(x) - \hat{f}(x))^2 + 2\epsilon (f(x) - \hat{f}(x)) + \epsilon^2 \right]\\ & = E\left[(f(x) - \hat{f}(x))^2 \right] + 2E(\epsilon) E(f(x) - \hat{f}(x)) + E(\epsilon^2)\\ & = (f(x) - \hat{f}(x))^2 + Var(\epsilon). \end{align*}

$\endgroup$
2
  • $\begingroup$ You're saying that given $X = x$, we are actually stating $E(Y - \hat{Y} | X = x) = (f(x) - \hat{f}(x))^2 + Var(\epsilon)$? $\endgroup$
    – Zhulu
    Apr 22 '20 at 1:43
  • $\begingroup$ Also, what's an intuitive way to understand the relationship of $Y|X$? For reference, the original ISLR (2017) pp. 19 states $Y = f(X) + \epsilon$, where $X$ is a set of predictors and $\epsilon$ is a random error term w/ mean 0. Why is $Y$ not a RV? $\endgroup$
    – Zhulu
    Apr 22 '20 at 1:53
1
$\begingroup$

Adding to @Greenparker, the first term of the last line is decomposed as below

$=(f(x)-\hat{f(x)}+E(\hat{f(x)})-E(\hat{f(x)})^2+Var(\epsilon)$

$=(\underbrace{E(\hat{f(x)})-f(x)}_{Bias})^2+(\underbrace{E(\hat{f(x)})-\hat{f(x)})^2}_{Var(\hat{f(x))}}+Var(\epsilon)$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.