# Expressing ridge solutions as sum of vectors

I'm having trouble following the derivation of the ridge solutions as a sum of vectors in formula 3.47 of The Elements of Statistical Learning on pg. 66. It shows the following:

$\mathbf{X}\hat{\beta}^{ridge} = \mathbf{X}(\mathbf{X}^T\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^T\mathbf{y}$

$\mathbf{X}\hat{\beta}^{ridge} = \mathbf{U}\mathbf{D}(\mathbf{D}^2 + \lambda\mathbf{I})^{-1}\mathbf{D}\mathbf{U}^T\mathbf{y}$

$\mathbf{X}\hat{\beta}^{ridge} = \sum{\mathbf{u_j}\frac{d_j^2}{d_j^2 + \lambda}\mathbf{u_j}^T\mathbf{y}}$

where $\mathbf{X} = \mathbf{UDV}^T$

I'm having trouble seeing how to get to line 2 after simplifying line 1 with $\mathbf{X}\hat{\beta}^{ridge} = \mathbf{UDV}^T(\mathbf{VDU}^T\mathbf{UDV} + \lambda\mathbf{I})^{-1}\mathbf{VDU}^T\mathbf{y}$

and then:

$\mathbf{X}\hat{\beta}^{ridge} = \mathbf{UDV}^T(\mathbf{VD ^2V} + \lambda\mathbf{I})^{-1}\mathbf{VDU}^T\mathbf{y}$ because $\mathbf{U}^T \mathbf{U} = \mathbf{I}$

Also, is there a general theorem that helps yield line 3? I can't wrap my head around going from matrix products to vector sums.

It's been a decade since I've taken linear algebra, so if you have any good resources for dealing with this type of math, please let me know.

From,

$$X\hat\beta^{ridge} = UDV^T(VD^2V^T + \lambda I)^{-1} VDU^Ty$$

(note the transpose in one of the $V$ inside the inverse) you get

$$X\hat\beta^{ridge} = UDV^T[V(D^2 + \lambda I)V^T]^{-1} VDU^Ty$$

(notice that $V(\lambda I)V^T = \lambda I$ as $V$ is orthogonal). Taking the inverse in the second equation we have, since $V^{-1} = V^T$ by orthogonality, that:

$$X\hat\beta^{ridge} = UDV^TV(D^2 + \lambda I)^{-1}V^TVDU^Ty$$

whence, again using the orthogonality of $V$, we get your second equation. Your third equation follows immediately as both $D$ and $D^2 + \lambda I$ are diagonal.

• Thanks! Sorry, I don't quite see how to manipulate the matrix multiplication into summations. If there's a good reference for turning matrix multiplication into summations, I'm happy to read it.
– Alec
Feb 6, 2012 at 7:09
• A diagonal matrix $D$ times a general matrix $P$ is just the sum of the diagonal elements of $D$ times the row vectors of $P$. In turn, $PD$ would is the sum of the diagonal elements of $D$ times the column vectors of $P$. Just write a simple example of low dimension, with matrices 3x3 for instance, and you will see at once how it goes. Feb 6, 2012 at 13:04