2
$\begingroup$

I'm having trouble following the derivation of the ridge solutions as a sum of vectors in formula 3.47 of The Elements of Statistical Learning on pg. 66. It shows the following:

$\mathbf{X}\hat{\beta}^{ridge} = \mathbf{X}(\mathbf{X}^T\mathbf{X} + \lambda\mathbf{I})^{-1}\mathbf{X}^T\mathbf{y}$

$\mathbf{X}\hat{\beta}^{ridge} = \mathbf{U}\mathbf{D}(\mathbf{D}^2 + \lambda\mathbf{I})^{-1}\mathbf{D}\mathbf{U}^T\mathbf{y}$

$\mathbf{X}\hat{\beta}^{ridge} = \sum{\mathbf{u_j}\frac{d_j^2}{d_j^2 + \lambda}\mathbf{u_j}^T\mathbf{y}}$

where $\mathbf{X} = \mathbf{UDV}^T$

I'm having trouble seeing how to get to line 2 after simplifying line 1 with $\mathbf{X}\hat{\beta}^{ridge} = \mathbf{UDV}^T(\mathbf{VDU}^T\mathbf{UDV} + \lambda\mathbf{I})^{-1}\mathbf{VDU}^T\mathbf{y}$

and then:

$\mathbf{X}\hat{\beta}^{ridge} = \mathbf{UDV}^T(\mathbf{VD ^2V} + \lambda\mathbf{I})^{-1}\mathbf{VDU}^T\mathbf{y}$ because $\mathbf{U}^T \mathbf{U} = \mathbf{I}$

Also, is there a general theorem that helps yield line 3? I can't wrap my head around going from matrix products to vector sums.

It's been a decade since I've taken linear algebra, so if you have any good resources for dealing with this type of math, please let me know.

$\endgroup$

1 Answer 1

1
$\begingroup$

From,

$$X\hat\beta^{ridge} = UDV^T(VD^2V^T + \lambda I)^{-1} VDU^Ty$$

(note the transpose in one of the $V$ inside the inverse) you get

$$X\hat\beta^{ridge} = UDV^T[V(D^2 + \lambda I)V^T]^{-1} VDU^Ty$$

(notice that $V(\lambda I)V^T = \lambda I$ as $V$ is orthogonal). Taking the inverse in the second equation we have, since $V^{-1} = V^T$ by orthogonality, that:

$$X\hat\beta^{ridge} = UDV^TV(D^2 + \lambda I)^{-1}V^TVDU^Ty$$

whence, again using the orthogonality of $V$, we get your second equation. Your third equation follows immediately as both $D$ and $D^2 + \lambda I$ are diagonal.

$\endgroup$
2
  • $\begingroup$ Thanks! Sorry, I don't quite see how to manipulate the matrix multiplication into summations. If there's a good reference for turning matrix multiplication into summations, I'm happy to read it. $\endgroup$
    – Alec
    Feb 6, 2012 at 7:09
  • 1
    $\begingroup$ A diagonal matrix $D$ times a general matrix $P$ is just the sum of the diagonal elements of $D$ times the row vectors of $P$. In turn, $PD$ would is the sum of the diagonal elements of $D$ times the column vectors of $P$. Just write a simple example of low dimension, with matrices 3x3 for instance, and you will see at once how it goes. $\endgroup$
    – F. Tusell
    Feb 6, 2012 at 13:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.