The book Elements of Statistical Learning (available in PDF online) discusses the optimisim bias (7.21, page 229). It states that the optimism bias is the difference between the training error and the in-sample error (error observed if we sample new outcome values at each of the original training points) (per below).

enter image description here

Next, it states this optimism bias ($\omega$) is equal to the covariance of our estimated y values and the actual y values (formula per below). I have trouble understanding why this formula indicates the optimism bias; naively i would have thought that a strong covariance between actual $y$ and predicted $y$ merely describes accuracy - not optimism. Let me know if someone can help with the derivation of the formula or share the intuition.

enter image description here

  • $\begingroup$ Very helpful, thank you! I think one of the equations has a minor typo and should be: $= {1 \over N}\sum_{i=1}^N \left( E_y[y_i^2] + E_y[\hat{y}_i^2] -2 E_y [y_i] E_y[ \hat{y}_i] - E_y[y_i^2] - E_y[\hat{y}_i^2] + 2E[y_i \hat{y}_i] \right)$ $\endgroup$
    – Sleepster
    Apr 14 '18 at 8:06

Let's start with the intuition.

There's nothing wrong with using $y_i$ to predict $\hat{y}_i$. In fact, not using it would mean we are throwing away valuable information. However the more we depend on the information contained in $y_i$ to come up with our prediction, the more overly optimistic our estimator will be.

On one extreme, if $\hat{y}_i$ is just $y_i$, you'll have perfect in sample prediction ($R^2 = 1$), but we're pretty sure the out-of-sample prediction is gonna be bad. In this case (it's easy to check by yourself), the degrees of freedom will be $df(\hat{y}) = n$.

On the other extreme, if you use the sample mean of $y$: $y_i = \hat{y_i} = \bar{y}$ for all $i$, then your degrees of freedom will just be 1.

Check this nice handout by Ryan Tibshirani for more details on this intuition

Now a similar proof to the other answer, but with a bit more explanation

Remember that, by definition, the average optimism is:

$$ \omega = E_y (Err_{in} - \overline{err}) $$

$$ = E_y \left( {1 \over N} \sum_{i=1}^N E_{Y^0} \left[ L(Y_i^0, \hat{f} (x_i) \; |\; T) \right] - {1 \over N} \sum_{i=1}^N L(y_i, \hat{f} (x_i) ) \right)$$

Now use a quadratic loss function and expand the squared terms:

$$ = E_y \left( {1 \over N} \sum_{i=1}^N E_{Y^0} \left[ (Y_i^0 - \hat{y}_i)^2 \right] - {1 \over N} \sum_{i=1}^N (y_i - \hat{y}_i)^2 ) \right)$$

$$ = {1 \over N} \sum_{i=1}^N\left( E_y E_{Y^0}[(Y_i^0)^2] + E_y E_{Y^0} [\hat{y}_i^2] -2 E_y E_{Y^0} [Y_i^0 \hat{y}_i] - E_y[y_i^2] - E_y[\hat{y}_i^2] + 2E[y_i \hat{y}_i] \right)$$

use $E_y E_{Y^0}[(Y_i^0)^2] = E_y[y_i^2]$ to replace:

$$ = {1 \over N}\sum_{i=1}^N \left( E_y[y_i^2] + E_y[\hat{y_i}^2] -2 E_y [y_i] E_y[ \hat{y}_i] - E_y[y_i^2] - E_y[\hat{y}_i^2] + 2E[y_i \hat{y}_i] \right)$$

$$ = {2 \over N} \sum_{i=1}^N \left( E[y_i \hat{y}_i] - E_y [y_i] E_y[ \hat{y}_i] \right)$$

To finish, note that $Cov(x, w) = E[xw] - E[x]E[w]$, which yields:

$$ = {2 \over N} \sum_{i=1}^N Cov(y_i, \hat{y}_i) $$

  • 5
    $\begingroup$ I have to point out that his name is spelled "Ryan Tibshirani" Rob Tibshirani $\endgroup$ Feb 14 '17 at 17:53
  • 2
    $\begingroup$ Welcome to our site, Rob--it is a privilege to have you here, if only for correcting an error! If you see any more, please let us know: and of course we would be glad of any answers you (or your students) might care to post. Your work is extensively referenced on this site, particularly ESL and Intro to the Bootstrap. $\endgroup$
    – whuber
    Feb 14 '17 at 18:22
  • $\begingroup$ Mind explaining $E_y E_{Y^0}[(Y_i^0)^2] = E_y[y_i^2]$? Also, is $2 E_y E_{Y^0} [Y_i^0 \hat{y}_i]=2 E_y [E_{Y^0} [Y_i^0]E_{Y^0}[\hat{y}_i]]=2 E_y [y_i] E_y[ \hat{y}_i]$? $\endgroup$
    – matanc1
    May 25 '18 at 20:36
  • $\begingroup$ @Shookie, $y$ and $Y^0$ are both for the distribution of the outcome given the same predicators. So they follow the same distribution and are independent of each other. So $E_yE_{Y^0} (Y_i^0)^2 = E_{Y^0} (Y_i^0)^2 = E_y y_i^2$. Their difference is that $\hat{y}_i$ is affected by $y$ but not $Y^0$. $y$ belongs to the training set and $Y^0$ belongs to the test set. $\endgroup$
    – doraemon
    Jul 4 '20 at 14:16

Let $\hat{f}(x_i)=\hat{y}_i$, then $$ \begin{aligned} \omega &= E_\boldsymbol{y}[op]\\ &=E_\boldsymbol{y}[Err_{in}-\overline{err}]\\ &=E_\boldsymbol{y}[Err_{in}]-E_\boldsymbol{y}[\overline{err}]\\ &=E_\boldsymbol{y}[\frac{1}{N}\sum_{i=1}^{N}E_{Y^0}[L(Y_i^0,\hat{f}(x_i))]-E_\boldsymbol{y}[\frac{1}{N}\sum_{i=1}^{N}L(y_i,\hat{f}(x_i))]\\ &=\frac{1}{N}\sum_{i=1}^{N}E_\boldsymbol{y}E_{Y^0}[(Y_i^0-\hat{y}_i)^2]-E_\boldsymbol{y}[(y_i-\hat{y}_i)^2]\\ &=\frac{1}{N}\sum_{i=1}^{N}E_\boldsymbol{y}E_{Y^0}[({Y_i^0})^2]+E_\boldsymbol{y}E_{Y^0}[{\hat{y}_i}^2]-2E_\boldsymbol{y}E_{Y^0}[Y_i^0\hat{y}_i]-E_\boldsymbol{y}[y_i^2]-E_\boldsymbol{y}[\hat{y}_i^2]+2E_\boldsymbol{y}[y_i\hat{y}_i]\\ &=\frac{1}{N}\sum_{i=1}^{N}E_\boldsymbol{y}[y_i^2]+E_\boldsymbol{y}[\hat{y}_i^2]-2E_\boldsymbol{y}[y_i]E_\boldsymbol{y}[\hat{y}_i]-E_\boldsymbol{y}[y_i^2]-E_\boldsymbol{y}[\hat{y}_i^2]+2E_\boldsymbol{y}[y_i\hat{y}_i]\\ &=\frac{2}{N}\sum_{i=1}^{N}E_\boldsymbol{y}[y_i\hat{y}_i]-E_\boldsymbol{y}[y_i]E_\boldsymbol{y}[\hat{y}_i]\\ &=\frac{2}{N}\sum_{i=1}^{N}E_\boldsymbol{y}[y_i\hat{y}_i-y_iE_\boldsymbol{y}[\hat{y}_i]-E_\boldsymbol{y}[y_i]\hat{y}_i+E_\boldsymbol{y}[y_i]E_\boldsymbol{y}[\hat{y}_i]]\\ &=\frac{2}{N}\sum_{i=1}^{N}E_\boldsymbol{y}[(\hat{y}_i-E_\boldsymbol{y}[\hat{y}_i])([y_i-E_\boldsymbol{y}[y_i])]\\ &=\frac{2}{N}\sum_{i=1}^{N}cov(\hat{y}_i,y_i) \end{aligned} $$ Q.E.D.

  • 1
    $\begingroup$ The last four steps can be simplified by this property of covariance: $E[x w ] - E[x] E[w] = Cov(x, w)$ $\endgroup$
    – cd98
    Oct 17 '16 at 22:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.