Kernel trick implemented for Ridge Regression

Question

I am trying to see the kernel trick implemented for Ridge Regression. As a first step, I want to rewrite the solution of Ridge regression. I know that:

$ \hat{\beta} = (X^TX + \lambda I_n)^{-1} X^T Y $

I think that if I can reach $ \hat{\beta} = X^T (XX^T + \lambda I_n)^{-1} Y = X^T p $

where $p = (XX^T + \lambda I_n)^{-1} Y $

I will be able to show the criterion minimized by RR and write it interms of $XX^T$

which thereafter can be replaced by the kernel operator K.

In this way I will prove how the kernel is used to calculate the inner product $XX^T$ without even visiting it.

So my question is: Can someone kindly guide me of how to go from $ \hat{\beta} = (X^T X + \lambda I_n)^{-1} X^T Y $

to $ \hat{\beta} = X^T (XX^T + \lambda I_n)^{-1} Y$

I have seen it in many books, but I didn't grasp the flow of steps. Can someone please help me with how to start this proof.

Answer 1

We can use the following matrix identity ($P_{n\times m}, Q_{m\times n}$):

$$(PQ+I)^{-1}P=P(QP+I)^{-1}$$

This can easily be verified by multiplying the equation by $(PQ+I)$ from left and $(QP+I)$ from right, which yields: $$P(QP+I)=(PQ+I)P\rightarrow PQP+P=PQP+P$$

We can use this identity with $\lambda I$ as well. So, $$\hat\beta=(\underbrace{X^T}_P\underbrace{X}_Q+\lambda I)^{-1}X^Ty=X^T(XX^T+\lambda I)^{-1}y$$