In section 7.3 of 'The Elements of Statistical Learning', the authors have shown the expression for bias-variance decomposition of linear model fit: enter image description here

enter image description here

However, I get a slightly different expression for $Err(x_0)$. My attempt to derivation is as follows: the standard bias-variance decomposition is $$Err(x_0)=\sigma_\epsilon^2+[f(x_0)-E_\tau\hat f_p(x_0)]^2+var(\hat f_p(x_0)),$$ where the subscript $\tau$ indicates that the expectation is w.r.t. training data and $$var(\hat f_p(x_0))=E_\tau[\hat f_p(x_0)-E_\tau\hat f_p(x_0)]^2.$$ Further, I can write: $$var(\hat f_p(x_0))=E_XE_{Y|X}[(\hat f_p(x_0)-E_\tau\hat f_p(x_0))^2|X].$$ Now, I take $$L=E_{Y|X}[(\hat f_p(x_0)-E_\tau\hat f_p(x_0))^2|X]$$ so that $var(\hat f_p(x_0))=E_X(L).$ On further simiplification: $$L=E_{Y|X}[x_0^T(X^TX)^{-1}X^T(y-\mu_{Y/X})(y-\mu_{Y/X})^TX(X^TX)^{-1}x_0|X],$$ $$=\sigma_\epsilon^2E_{Y|X}[x_0^T(X^TX)^{-1}X^TI_NX(X^TX)^{-1}x_0|X],$$ $$=\sigma_\epsilon^2E_{Y|X}[||h(x_0)||^2|X],$$ where $$h(x_0)=X(X^TX)^{-1}x_0.$$ Now $$var(\hat f_p(x_0))=\sigma_\epsilon^2E_XE_{Y|X}[||h(x_0)||^2|X],$$ $$=\sigma_\epsilon^2E[||h(x_0)||^2].$$

However, in the book $var(\hat f_p(x_0))=||h(x_0)||^2\sigma_\epsilon^2.$ Is there anything wrong with my derivation or how can I proceed further to remove the extra expectation operator.


1 Answer 1


You should drop the outside $E_X$, and only use $E_{Y|X}$, because here they are computing the test error (definition is at the beginning of this chapter eq 7.2), and the training sample X is fixed(you add $E_X$ when you want to compute the expected test error in eq 7.3), only the response variable Y is random (due to $\epsilon$). This is the meaning of the condition X. When you put in $x_0$, you can think it is a new number but also a 'fixed' number, so the "random" part only comes from $Y=AX + \epsilon$, the irreducible error.

In 7.12, they are giving the result as training error, and that's why $p/N \sigma^2_{\epsilon}$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.