Skip to main content
added 317 characters in body
Source Link
jld
  • 20.8k
  • 2
  • 64
  • 69

There's a matrix identity $$ PB^T(BPB^T + R)^{-1} = (P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1} $$ for matrices with concordant dimensions and invertibility.

Taking $P = I$, $R = \lambda I$, $B^T = H$, and $B = H^T$, we have $$ H(H^TH + \lambda I)^{-1} = (\lambda I + HH^T)^{-1}H. $$ Let $$ h_x = \left(\begin{array}{c}h_1(x) \\ \vdots \\ h_M(x)\end{array}\right) $$ so we have $$ \hat f(x) = h_x^T\hat\beta = h_x^T(H^TH + \lambda I)^{-1}H^Ty \\ = h_x^TH^T(HH^T + \lambda I)^{-1}y \\ = \hat\alpha^T(Hh_x) $$ by the aforementioned identity. Now $$ (Hh_x)_i = \langle h_{x_i}, h_x\rangle = K(x, x_i) $$ so all together we have $$ \hat f(x) = \sum_{i=1}^N \hat\alpha_i K(x,x_i) $$


To prove that identity, you can note that $$ PB^T(BPB^T + R)^{-1} - (P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1} \\ = (P^{-1} + B^TR^{-1}B)^{-1}\left[(P^{-1} + B^TR^{-1}B)PB^T - B^TR^{-1}(BPB^T + R)\right](BPB^T + R)^{-1} \\ = C\left[B^T + B^TR^{-1}BPB^T - B^TR^{-1}BPB^T - B^T\right]D \\ = \mathbf 0 $$

There's a matrix identity $$ PB^T(BPB^T + R)^{-1} = (P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1} $$ for matrices with concordant dimensions and invertibility.

Taking $P = I$, $R = \lambda I$, $B^T = H$, and $B = H^T$, we have $$ H(H^TH + \lambda I)^{-1} = (\lambda I + HH^T)^{-1}H. $$ Let $$ h_x = \left(\begin{array}{c}h_1(x) \\ \vdots \\ h_M(x)\end{array}\right) $$ so we have $$ \hat f(x) = h_x^T\hat\beta = h_x^T(H^TH + \lambda I)^{-1}H^Ty \\ = h_x^TH^T(HH^T + \lambda I)^{-1}y \\ = \hat\alpha^T(Hh_x) $$ by the aforementioned identity. Now $$ (Hh_x)_i = \langle h_{x_i}, h_x\rangle = K(x, x_i) $$ so all together we have $$ \hat f(x) = \sum_{i=1}^N \hat\alpha_i K(x,x_i) $$

There's a matrix identity $$ PB^T(BPB^T + R)^{-1} = (P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1} $$ for matrices with concordant dimensions and invertibility.

Taking $P = I$, $R = \lambda I$, $B^T = H$, and $B = H^T$, we have $$ H(H^TH + \lambda I)^{-1} = (\lambda I + HH^T)^{-1}H. $$ Let $$ h_x = \left(\begin{array}{c}h_1(x) \\ \vdots \\ h_M(x)\end{array}\right) $$ so we have $$ \hat f(x) = h_x^T\hat\beta = h_x^T(H^TH + \lambda I)^{-1}H^Ty \\ = h_x^TH^T(HH^T + \lambda I)^{-1}y \\ = \hat\alpha^T(Hh_x) $$ by the aforementioned identity. Now $$ (Hh_x)_i = \langle h_{x_i}, h_x\rangle = K(x, x_i) $$ so all together we have $$ \hat f(x) = \sum_{i=1}^N \hat\alpha_i K(x,x_i) $$


To prove that identity, you can note that $$ PB^T(BPB^T + R)^{-1} - (P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1} \\ = (P^{-1} + B^TR^{-1}B)^{-1}\left[(P^{-1} + B^TR^{-1}B)PB^T - B^TR^{-1}(BPB^T + R)\right](BPB^T + R)^{-1} \\ = C\left[B^T + B^TR^{-1}BPB^T - B^TR^{-1}BPB^T - B^T\right]D \\ = \mathbf 0 $$

Source Link
jld
  • 20.8k
  • 2
  • 64
  • 69

There's a matrix identity $$ PB^T(BPB^T + R)^{-1} = (P^{-1} + B^TR^{-1}B)^{-1}B^TR^{-1} $$ for matrices with concordant dimensions and invertibility.

Taking $P = I$, $R = \lambda I$, $B^T = H$, and $B = H^T$, we have $$ H(H^TH + \lambda I)^{-1} = (\lambda I + HH^T)^{-1}H. $$ Let $$ h_x = \left(\begin{array}{c}h_1(x) \\ \vdots \\ h_M(x)\end{array}\right) $$ so we have $$ \hat f(x) = h_x^T\hat\beta = h_x^T(H^TH + \lambda I)^{-1}H^Ty \\ = h_x^TH^T(HH^T + \lambda I)^{-1}y \\ = \hat\alpha^T(Hh_x) $$ by the aforementioned identity. Now $$ (Hh_x)_i = \langle h_{x_i}, h_x\rangle = K(x, x_i) $$ so all together we have $$ \hat f(x) = \sum_{i=1}^N \hat\alpha_i K(x,x_i) $$