1
$\begingroup$

enter image description here

THis is from the Elements of Statistical Learning book page 437 in the section of support vector machine. Can anyone give me some hint for the missing derivation steps for why 12.49 is true (as seen in the attached screenshot)? I do not see why 12.48 gives rise to 12.49.

Many Thanks!

$\endgroup$
1
$\begingroup$

There's a matrix identity $$ PB^T(BPB^T + R)^{-1} = (P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1} $$ for matrices with concordant dimensions and invertibility.

Taking $P = I$, $R = \lambda I$, $B^T = H$, and $B = H^T$, we have $$ H(H^TH + \lambda I)^{-1} = (\lambda I + HH^T)^{-1}H. $$ Let $$ h_x = \left(\begin{array}{c}h_1(x) \\ \vdots \\ h_M(x)\end{array}\right) $$ so we have $$ \hat f(x) = h_x^T\hat\beta = h_x^T(H^TH + \lambda I)^{-1}H^Ty \\ = h_x^TH^T(HH^T + \lambda I)^{-1}y \\ = \hat\alpha^T(Hh_x) $$ by the aforementioned identity. Now $$ (Hh_x)_i = \langle h_{x_i}, h_x\rangle = K(x, x_i) $$ so all together we have $$ \hat f(x) = \sum_{i=1}^N \hat\alpha_i K(x,x_i) $$


To prove that identity, you can note that $$ PB^T(BPB^T + R)^{-1} - (P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1} \\ = (P^{-1} + B^TR^{-1}B)^{-1}\left[(P^{-1} + B^TR^{-1}B)PB^T - B^TR^{-1}(BPB^T + R)\right](BPB^T + R)^{-1} \\ = C\left[B^T + B^TR^{-1}BPB^T - B^TR^{-1}BPB^T - B^T\right]D \\ = \mathbf 0 $$

$\endgroup$
  • $\begingroup$ Many thanks to you jld for the detailed proof $\endgroup$ – humenghu Jul 12 '19 at 9:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.