1
$\begingroup$

I am trying to see the kernel trick implemented for Ridge Regression. As a first step, I want to rewrite the solution of Ridge regression. I know that:

$ \hat{\beta} = (X^TX + \lambda I_n)^{-1} X^T Y $

I think that if I can reach $ \hat{\beta} = X^T (XX^T + \lambda I_n)^{-1} Y = X^T p $

where $p = (XX^T + \lambda I_n)^{-1} Y $

I will be able to show the criterion minimized by RR and write it interms of $XX^T$

which thereafter can be replaced by the kernel operator K.

In this way I will prove how the kernel is used to calculate the inner product $XX^T$ without even visiting it.

So my question is: Can someone kindly guide me of how to go from $ \hat{\beta} = (X^T X + \lambda I_n)^{-1} X^T Y $

to $ \hat{\beta} = X^T (XX^T + \lambda I_n)^{-1} Y$

I have seen it in many books, but I didn't grasp the flow of steps. Can someone please help me with how to start this proof.

$\endgroup$
2
  • $\begingroup$ The solution of ridge regression should be $ (X^TX + \lambda I_n)^{-1} X^T y$ instead of $ (XX^T + \lambda I_n)^{-1} X^T y$ $\endgroup$
    – Bayesian
    Jun 17, 2022 at 6:41
  • $\begingroup$ you're right, thanks! $\endgroup$
    – user360583
    Jun 17, 2022 at 7:17

1 Answer 1

0
$\begingroup$

We can use the following matrix identity ($P_{n\times m}, Q_{m\times n}$):

$$(PQ+I)^{-1}P=P(QP+I)^{-1}$$

This can easily be verified by multiplying the equation by $(PQ+I)$ from left and $(QP+I)$ from right, which yields: $$P(QP+I)=(PQ+I)P\rightarrow PQP+P=PQP+P$$

We can use this identity with $\lambda I$ as well. So, $$\hat\beta=(\underbrace{X^T}_P\underbrace{X}_Q+\lambda I)^{-1}X^Ty=X^T(XX^T+\lambda I)^{-1}y$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.