I'm reading Elements of Statistical learning Chapter 7. I am slightly confused about one term in the derviation of the in sample error equation (equation 7.12) enter image description hereenter image description here

Can some explain/show mathematically how the variance term changes to become $\frac{p}{N}\sigma_{\epsilon}^2$ in equation 7.12? I know the $1/N$ is due to the fact that the second equation relates to the traning error, and so is an average loss over the training sample. However, I'm unsure where the $p$ comes from.


1 Answer 1


I'm pretty sure that this is the hard way, but here it is. First, simplify the following: $$||\mathbf h(x_i)||^2 = \mathbf h(x_i)^T \mathbf h(x_i) = x_i^T \mathbf (\mathbf X^T\mathbf X)^{-1}x_i$$

We want this term to be summed over all $i$. Let the SVD of $\mathbf X^T\mathbf X$ be $\mathbf U\mathbf S\mathbf U^T$ (since it's symmetric and positive semi-definite, this is basically the diagonalised version of it and $\mathbf U\mathbf U^T=\mathbf I$, columns of $\mathbf U$ are orthonormal). Then, $$(\mathbf X^T\mathbf X)^{-1}=\mathbf U\mathbf S^{-1}\mathbf U^T=\sum_{j=1}^p\frac{1}{\sigma_j} u_ju_j^T$$, where $\sigma_j$ are the diagonal entries of $\mathbf S$. Substitute it in the sum:

$$\begin{align}\sum_{i=1}^n ||\mathbf h (x_i)||^2&=\sum_{i=1}^nx_i^T\left(\sum_{j=1}^p\frac{1}{\sigma_j} u_ju_j^T \right)x_i=\sum_{j=1}^p\sum_{i=1}^n\frac{1}{\sigma_j} x_i^Tu_j u_j^Tx_i\\&=\sum_{j=1}^p\sum_{i=1}^n\frac{1}{\sigma_j} u_j^Tx_ix_i^Tu_j^T=\sum_{j=1}^p\frac{1}{\sigma_j} u_j^T\left(\sum_{i=1}^n x_ix_i^T\right) u_j\\&=\sum_{j=1}^p\frac{1}{\sigma_j} u_j^T(\mathbf X^T\mathbf X)u_j=\sum_{j=1}^p\frac{1}{\sigma_j} u_j^T\left(\sum_{k=1}^p \sigma_ku_ku_k^T\right) u_j\\&=\sum_{j=1}^p\sum_{k=1}^p \frac{\sigma_k}{\sigma_j}(u_j^Tu_k)(u_k^Tu_j)\end{align}$$

Since $u_k^Tu_j=0$ for $k\neq j$, and $1$ if $k=j$, the inside sum is $1$, which yields the total sum $p$.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.