THis is from the Elements of Statistical Learning book page 437 in the section of support vector machine. Can anyone give me some hint for the missing derivation steps for why 12.49 is true (as seen in the attached screenshot)? I do not see why 12.48 gives rise to 12.49.
Many Thanks!
There's a matrix identity $$ PB^T(BPB^T + R)^{-1} = (P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1} $$ for matrices with concordant dimensions and invertibility.
Taking $P = I$, $R = \lambda I$, $B^T = H$, and $B = H^T$, we have $$ H(H^TH + \lambda I)^{-1} = (\lambda I + HH^T)^{-1}H. $$ Let $$ h_x = \left(\begin{array}{c}h_1(x) \\ \vdots \\ h_M(x)\end{array}\right) $$ so we have $$ \hat f(x) = h_x^T\hat\beta = h_x^T(H^TH + \lambda I)^{-1}H^Ty \\ = h_x^TH^T(HH^T + \lambda I)^{-1}y \\ = \hat\alpha^T(Hh_x) $$ by the aforementioned identity. Now $$ (Hh_x)_i = \langle h_{x_i}, h_x\rangle = K(x, x_i) $$ so all together we have $$ \hat f(x) = \sum_{i=1}^N \hat\alpha_i K(x,x_i) $$
To prove that identity, you can note that $$ PB^T(BPB^T + R)^{-1} - (P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1} \\ = (P^{-1} + B^TR^{-1}B)^{-1}\left[(P^{-1} + B^TR^{-1}B)PB^T - B^TR^{-1}(BPB^T + R)\right](BPB^T + R)^{-1} \\ = C\left[B^T + B^TR^{-1}BPB^T - B^TR^{-1}BPB^T - B^T\right]D \\ = \mathbf 0 $$
