In the book Elements of statistical learning 2 on page 229, they express the expected optimism of the training error as: $$ \omega=\frac{2}{N} \sum_{i=1}^{N} \operatorname{Cov}\left(\hat{y}_{i}, y_{i}\right) $$

This is followed by the following paragraph:

This expression simplifies if $\hat{y}_{i}$ is obtained by a linear fit with $d$ inputs or basis functions. For example, $$ \sum_{i=1}^{N} \operatorname{Cov}\left(\hat{y}_{i}, y_{i}\right)=d \sigma_{\varepsilon}^{2} $$ for the additive error model $Y=f(X)+\varepsilon$

I'm struggling to see where this relation comes from. Why is the sum of the covariances between predicted and true values equal to the the number of basis functions times the variance of the errors in the linear model? What are basis functions in this context, exactly?


Since we are in the case of a linear fit of the additive error model $$\hat{{y}} = X(X^TX)^{-1}X^Ty$$ And we notice that $\sum_{i=1}^{N}{cov(\hat{y}_i, y_i)}$ is just the sum of the diagonal elements of the covariance matrix $\Sigma = cov(\hat{y}, y)$. In other words
$$\sum_{i=1}^{N}{cov(\hat{y}_i, y_i)} = trace(\Sigma)$$

Putting these facts together we obtain $$\begin{aligned} trace(\Sigma) &= trace(cov(\mathbf{\hat{y}}, \mathbf{y})) \\ &= trace(cov(\mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T \mathbf{y}, \mathbf{y})) \\ &= trace(\mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^Tcov(\mathbf{y}, \mathbf{y})) \\ &= trace(\mathbf{X}(\mathbf{X}^T\mathbf{X})^{-1}\mathbf{X}^T)\sigma^2_\epsilon \\ &= trace((\mathbf{X}^T\mathbf{X})^{-1}(\mathbf{X}^T\mathbf{X}))\sigma^2_\epsilon \ \ \ \ \ \ \ \ \text{ (permuting inside the trace)} \\ &= trace(\mathbf{I}_d)\sigma^2_\epsilon \\ &= d\sigma^2_\epsilon \ \ \ \ \ \ \ \ \text{ (as required)} \\ \end{aligned} $$

And basis functions are just functions of the original input variables. Common examples are polynomial or interaction terms.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.