Section 7.3 of Elements of Statistical Learning (2nd edition) gives the bias-variance decomposition for OLS prediction first for a single input $x_0$, and then averaged over a set of inputs $x_1, \dots, x_N$.

I understand where the variance term $||h(x_0)||^2 \sigma_{\varepsilon}^2$ in equation 7.11 comes from, since we have $\hat{f}_p(x_0) = \text{A}y$ for some matrix $A$, and $\text{Var}(Ay) = A\text{Var}(y)A^T$.

However, I don't understand where $p$ comes from in equation 7.12. When I try to compute the average of variance terms, it seems to me that we have $$ \frac{1}{N} \left[ ||X(X^TX)^{-1}x_1||^2 + \dots + ||X(X^TX)^{-1}x_N||^2 \right] = \frac{1}{N} ||X(X^TX)^{-1}X^T \mathbb{1}||^2, $$ where $\mathbb{1}$ is the $N \times 1$ column vector of 1's.

Elements of Statistical Learning, Section 7.3


1 Answer 1


$\mathbf X$ can be expressed in the form of thin SVD (cf. $\rm [I]$) as $$\mathbf X= \mathbf U_p\boldsymbol\Sigma\mathbf V^\top.\tag 1\label 1$$

$$\therefore~~ (\mathbf X^\top\mathbf X)^{-1} = \mathbf V\mathbf \Sigma^{-1}\mathbf\Sigma^{-1}\mathbf V^\top = \sum_{k~=~1}^p \frac{\mathbf v_k \mathbf v_k^\top}{\lambda_k}, \tag 2\label 2$$ where $\mathbf v_k$ are the columns of $\bf V$ and $\lambda_k$ are the diagonal elements of $\mathbf \Lambda:= \mathbf \Sigma^2.$


\begin{align}\sum_{i~=~1}^N \Vert\mathbf h(\mathbf x_i)\Vert^2 &= \sum_{i~=~1}^N \mathbf x_i^\top (\mathbf X^\top\mathbf X)^{-1}\mathbf x_i\\ &\overset{\eqref 2}{=} \sum_{i~=~1}^N\sum_{k~=~1}^p \frac{\mathbf x_i^\top\mathbf v_k \mathbf v_k^\top\mathbf x_i}{\lambda_k}\\ &\overset{\eqref 1}{=} \sum_{k~=~1}^p\frac{\mathbf v_k^\top \left(\displaystyle\sum_{l~=~1}^p \mathbf v_l \lambda_l\mathbf v_l^\top \right)\mathbf v_k}{\lambda_k}\\ &= \sum_{k~=~1}^p \sum_{l~=~1}^p \frac{\mathbf v_k^\top \mathbf v_l \lambda_l\mathbf v_l^\top\mathbf v_k}{\lambda_k}\\ &= p. \tag 3 \end{align}


$\rm [I]$ Linear Regression analysis, George A. F. Seber, Alan J. Lee, John Wiley & sons., $2003,$ sec. $11.4,$ pp. $353-354.$


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.