# Kernel trick implemented for Ridge Regression

I am trying to see the kernel trick implemented for Ridge Regression. As a first step, I want to rewrite the solution of Ridge regression. I know that:

$$\hat{\beta} = (X^TX + \lambda I_n)^{-1} X^T Y$$

I think that if I can reach $$\hat{\beta} = X^T (XX^T + \lambda I_n)^{-1} Y = X^T p$$

where $$p = (XX^T + \lambda I_n)^{-1} Y$$

I will be able to show the criterion minimized by RR and write it interms of $$XX^T$$

which thereafter can be replaced by the kernel operator K.

In this way I will prove how the kernel is used to calculate the inner product $$XX^T$$ without even visiting it.

So my question is: Can someone kindly guide me of how to go from $$\hat{\beta} = (X^T X + \lambda I_n)^{-1} X^T Y$$

to $$\hat{\beta} = X^T (XX^T + \lambda I_n)^{-1} Y$$

I have seen it in many books, but I didn't grasp the flow of steps. Can someone please help me with how to start this proof.

• The solution of ridge regression should be $(X^TX + \lambda I_n)^{-1} X^T y$ instead of $(XX^T + \lambda I_n)^{-1} X^T y$ Jun 17, 2022 at 6:41
• you're right, thanks! Jun 17, 2022 at 7:17

We can use the following matrix identity ($$P_{n\times m}, Q_{m\times n}$$):
$$(PQ+I)^{-1}P=P(QP+I)^{-1}$$
This can easily be verified by multiplying the equation by $$(PQ+I)$$ from left and $$(QP+I)$$ from right, which yields: $$P(QP+I)=(PQ+I)P\rightarrow PQP+P=PQP+P$$
We can use this identity with $$\lambda I$$ as well. So, $$\hat\beta=(\underbrace{X^T}_P\underbrace{X}_Q+\lambda I)^{-1}X^Ty=X^T(XX^T+\lambda I)^{-1}y$$