bias-variance decomposition for OLS

Question

Section 7.3 of Elements of Statistical Learning (2nd edition) gives the bias-variance decomposition for OLS prediction first for a single input $x_0$, and then averaged over a set of inputs $x_1, \dots, x_N$.

I understand where the variance term $||h(x_0)||^2 \sigma_{\varepsilon}^2$ in equation 7.11 comes from, since we have $\hat{f}_p(x_0) = \text{A}y$ for some matrix $A$, and $\text{Var}(Ay) = A\text{Var}(y)A^T$.

However, I don't understand where $p$ comes from in equation 7.12. When I try to compute the average of variance terms, it seems to me that we have $$ \frac{1}{N} \left[ ||X(X^TX)^{-1}x_1||^2 + \dots + ||X(X^TX)^{-1}x_N||^2 \right] = \frac{1}{N} ||X(X^TX)^{-1}X^T \mathbb{1}||^2, $$ where $\mathbb{1}$ is the $N \times 1$ column vector of 1's.

Answer 1

$\mathbf X$ can be expressed in the form of thin SVD (cf. $\rm [I]$) as $$\mathbf X= \mathbf U_p\boldsymbol\Sigma\mathbf V^\top.\tag 1\label 1$$

$$\therefore~~ (\mathbf X^\top\mathbf X)^{-1} = \mathbf V\mathbf \Sigma^{-1}\mathbf\Sigma^{-1}\mathbf V^\top = \sum_{k~=~1}^p \frac{\mathbf v_k \mathbf v_k^\top}{\lambda_k}, \tag 2\label 2$$ where $\mathbf v_k$ are the columns of $\bf V$ and $\lambda_k$ are the diagonal elements of $\mathbf \Lambda:= \mathbf \Sigma^2.$

Now,

\begin{align}\sum_{i~=~1}^N \Vert\mathbf h(\mathbf x_i)\Vert^2 &= \sum_{i~=~1}^N \mathbf x_i^\top (\mathbf X^\top\mathbf X)^{-1}\mathbf x_i\\ &\overset{\eqref 2}{=} \sum_{i~=~1}^N\sum_{k~=~1}^p \frac{\mathbf x_i^\top\mathbf v_k \mathbf v_k^\top\mathbf x_i}{\lambda_k}\\ &\overset{\eqref 1}{=} \sum_{k~=~1}^p\frac{\mathbf v_k^\top \left(\displaystyle\sum_{l~=~1}^p \mathbf v_l \lambda_l\mathbf v_l^\top \right)\mathbf v_k}{\lambda_k}\\ &= \sum_{k~=~1}^p \sum_{l~=~1}^p \frac{\mathbf v_k^\top \mathbf v_l \lambda_l\mathbf v_l^\top\mathbf v_k}{\lambda_k}\\ &= p. \tag 3 \end{align}

---

Reference:

$\rm [I]$ Linear Regression analysis, George A. F. Seber, Alan J. Lee, John Wiley & sons., $2003,$ sec. $11.4,$ pp. $353-354.$