17
$\begingroup$

I am reading the chapter of bias-variance tradeoff of The elements of statistical learning and I have doubt in the formula at page 29. Let the data arise from a model such that $$ Y = f(x)+\epsilon$$ where $\epsilon$ is random number with expected value $\hat{\epsilon} = E[\epsilon]=0$ and Variance $E[(\epsilon - \hat\epsilon)^2]=E[\epsilon^2]=\sigma^2$. Let the expected value of error of the model is $$ E[(Y-f_k(x))^2] $$ where $f_k(x)$ is the prediction of $x$ of our learner. According to the book, the error is $$ E[(Y-f_k(x))^2]=\sigma^2+Bias(f_k)^2+Var(f_k(x)). $$

My question is why bias term is not 0? developing the formula of the error I see $$ E[(Y-f_k(x))^2]=\\ E[(f(x)+\epsilon-f_k(x))^2]=\\ E[(f(x)-f_k(x))^2]+2E[(f(x)-f_k(x))\epsilon]+E[\epsilon^2]=\\ Var(f_k(x))+2E[(f(x)-f_k(x))\epsilon]+\sigma^2 $$

as $\epsilon$ is an independent random number $2E[(f(x)-f_k(x))\epsilon]=2E[(f(x)-f_k(x))]E[\epsilon]=0$

Where I am wrong?

$\endgroup$
17
$\begingroup$

You are not wrong, but you made an error in one step since $E[(f(x)-f_k(x))^2] \ne Var(f_k(x))$. $E[(f(x)-f_k(x))^2]$ is $\text{MSE}(f_k(x)) = Var(f_k(x)) + \text{Bias}^2(f_k(x))$.

\begin{align*} E[(Y-f_k(x))^2]& = E[(f(x)+\epsilon-f_k(x))^2] \\ &= E[(f(x)-f_k(x))^2]+2E[(f(x)-f_k(x))\epsilon]+E[\epsilon^2]\\ &= E\left[\left(f(x) - E(f_k(x)) + E(f_k(x))-f_k(x) \right)^2 \right] + 2E[(f(x)-f_k(x))\epsilon]+\sigma^2 \\ & = Var(f_k(x)) + \text{Bias}^2(f_k(x)) + \sigma^2. \end{align*}

Note: $E[(f_k(x)-E(f_k(x)))(f(x)-E(f_k(x))] = E[f_k(x)-E(f_k(x))](f(x)-E(f_k(x))) = 0.$

$\endgroup$
  • $\begingroup$ In case of binary outcomes, Is there an equivalent proof with cross entropy as error measure? $\endgroup$ – emanuele Mar 28 '16 at 15:22
  • 1
    $\begingroup$ It doesn't work out quite so well with a binary response. See Ex 7.2 in the second edition of "The Elements of Statistical Learning". $\endgroup$ – Matthew Drury Mar 28 '16 at 21:25
  • 2
    $\begingroup$ could you explain how you go from $E\left[\left(f(x) - E(f_k(x)) + E(f_k(x))-f_k(x) \right)^2 \right] + 2E[(f(x)-f_k(x))\epsilon]+\sigma^2$ to $Var(f_k(x)) + \text{Bias}^2(f_k(x)) + \sigma^2$? $\endgroup$ – Antoine Apr 30 '18 at 14:31
10
$\begingroup$

A few more steps of the Bias - Variance decomposition

Indeed, the full derivation is rarely given in textbooks as it involves a lot of uninspiring algebra. Here is a more complete derivation using notation from the book "Elements of Statistical Learning" on page 223


If we assume that $Y = f(X) + \epsilon$ and $E[\epsilon] = 0$ and $Var(\epsilon) = \sigma^2_\epsilon$ then we can derive the expression for the expected prediction error of a regression fit $\hat f(X)$ at an input $X = x_0$ using squared error loss

$$Err(x_0) = E[ (Y - \hat f(x_0) )^2 | X = x_0]$$

For notational simplicity let $\hat f(x_0) = \hat f$, $f(x_0) = f$ and recall that $E[f] = f$ and $E[Y] = f$

\begin{aligned} E[ (Y - \hat f)^2 ] &= E[(Y - f + f - \hat f )^2] \\ & = E[(y - f)^2] + E[(f - \hat f)^2] + 2 E[(f - \hat f)(y - f)] \\ & = E[(f + \epsilon - f)^2] + E[(f - \hat f)^2] + 2E[fY - f^2 - \hat f Y + \hat f f] \\ & = E[\epsilon^2] + E[(f - \hat f)^2] + 2( f^2 - f^2 - f E[\hat f] + f E[\hat f] ) \\ & = \sigma^2_\epsilon + E[(f - \hat f)^2] + 0 \end{aligned}

For the term $E[(f - \hat f)^2]$ we can use a similar trick as above, adding and subtracting $E[\hat f]$ to get

\begin{aligned} E[(f - \hat f)^2] & = E[(f + E[\hat f] - E[\hat f] - \hat f)^2] \\ & = E \left[ f - E[\hat f] \right]^2 + E\left[ \hat f - E[ \hat f] \right]^2 \\ & = \left[ f - E[\hat f] \right]^2 + E\left[ \hat f - E[ \hat f] \right]^2 \\ & = Bias^2[\hat f] + Var[\hat f] \end{aligned}

Putting it together

$$E[ (Y - \hat f)^2 ] = \sigma^2_\epsilon + Bias^2[\hat f] + Var[\hat f] $$


Some comments on why $E[\hat f Y] = f E[\hat f]$

Taken from Alecos Papadopoulos here

Recall that $\hat f$ is the predictor we have constructed based on the $m$ data points $\{(x^{(1)},y^{(1)}),...,(x^{(m)},y^{(m)}) \}$ so we can write $\hat f = \hat f_m$ to remember that.

On the other hand $Y$ is the prediction we are making on a new data point $(x^{(m+1)},y^{(m+1)})$ by using the model constructed on the $m$ data points above. So the Mean Squared Error can be written as

$$ E[\hat f_m(x^{(m+1)}) - y^{(m+1)}]^2$$

Expanding the equation from the previous section

$$E[\hat f_m Y]=E[\hat f_m (f+ \epsilon)]=E[\hat f_m f+\hat f_m \epsilon]=E[\hat f_m f]+E[\hat f_m \epsilon]$$

The last part of the equation can be viewed as

$$ E[\hat f_m(x^{(m+1)}) \cdot \epsilon^{(m+1)}] = 0$$

Since we make the following assumptions about the point $x^{(m+1)}$:

  • It was not used when constructing $\hat f_m$
  • It is independent of all other observations $\{(x^{(1)},y^{(1)}),...,(x^{(m)},y^{(m)}) \}$
  • It is independent of $\epsilon^{(m+1)}$

Other sources with full derivations

$\endgroup$
  • 1
    $\begingroup$ Why $E[\hat{f}Y]=f E[\hat{f}]$? I don't think $Y$ and $\hat{f}$ are independent, since $\hat{f}$ is essentially constructed using $Y$. $\endgroup$ – Felipe Pérez Sep 10 '18 at 11:53
  • 2
    $\begingroup$ But the question is essentially the same, why $E[\hat{f}\epsilon]=0$? The randomness of $\hat{f}$ comes from the error $\epsilon$ so I don't see why would $\hat{f}$ and $\epsilon$ be independent, and hence, $\mathbb{E}(\hat{f}\epsilon)=0$. $\endgroup$ – Felipe Pérez Sep 10 '18 at 12:20
  • $\begingroup$ From your precisation seems that the in sample vs out of sample perspective is crucial. It's so? If we work only in sample and, then, see $\epsilon$ as residual the bias variance tradeoff disappear? $\endgroup$ – markowitz May 15 at 7:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.