Bias-variance docomposition of linear model fit in 'The Elements of Statistical Learning'

Question

In section 7.3 of 'The Elements of Statistical Learning', the authors have shown the expression for bias-variance decomposition of linear model fit:

However, I get a slightly different expression for $Err(x_0)$. My attempt to derivation is as follows: the standard bias-variance decomposition is $$Err(x_0)=\sigma_\epsilon^2+[f(x_0)-E_\tau\hat f_p(x_0)]^2+var(\hat f_p(x_0)),$$ where the subscript $\tau$ indicates that the expectation is w.r.t. training data and $$var(\hat f_p(x_0))=E_\tau[\hat f_p(x_0)-E_\tau\hat f_p(x_0)]^2.$$ Further, I can write: $$var(\hat f_p(x_0))=E_XE_{Y|X}[(\hat f_p(x_0)-E_\tau\hat f_p(x_0))^2|X].$$ Now, I take $$L=E_{Y|X}[(\hat f_p(x_0)-E_\tau\hat f_p(x_0))^2|X]$$ so that $var(\hat f_p(x_0))=E_X(L).$ On further simiplification: $$L=E_{Y|X}[x_0^T(X^TX)^{-1}X^T(y-\mu_{Y/X})(y-\mu_{Y/X})^TX(X^TX)^{-1}x_0|X],$$ $$=\sigma_\epsilon^2E_{Y|X}[x_0^T(X^TX)^{-1}X^TI_NX(X^TX)^{-1}x_0|X],$$ $$=\sigma_\epsilon^2E_{Y|X}[||h(x_0)||^2|X],$$ where $$h(x_0)=X(X^TX)^{-1}x_0.$$ Now $$var(\hat f_p(x_0))=\sigma_\epsilon^2E_XE_{Y|X}[||h(x_0)||^2|X],$$ $$=\sigma_\epsilon^2E[||h(x_0)||^2].$$

However, in the book $var(\hat f_p(x_0))=||h(x_0)||^2\sigma_\epsilon^2.$ Is there anything wrong with my derivation or how can I proceed further to remove the extra expectation operator.

Answer 1

You should drop the outside $E_X$, and only use $E_{Y|X}$, because here they are computing the test error (definition is at the beginning of this chapter eq 7.2), and the training sample X is fixed(you add $E_X$ when you want to compute the expected test error in eq 7.3), only the response variable Y is random (due to $\epsilon$). This is the meaning of the condition X. When you put in $x_0$, you can think it is a new number but also a 'fixed' number, so the "random" part only comes from $Y=AX + \epsilon$, the irreducible error.

In 7.12, they are giving the result as training error, and that's why $p/N \sigma^2_{\epsilon}$.