We can kernelize Ridge regression as shown in these notes: https://www.ics.uci.edu/~welling/classnotes/papers_class/Kernel-Ridge.pdf.

However would it be possible to find a vector $\boldsymbol\alpha$ such that we can express linear regression as $$f(\mathbf x)=\sum_{i=1}^N \alpha_i \kappa(\mathbf x,\mathbf x_i)$$ where $\mathbf x\in \mathbb R^N$, and $\kappa:\mathbb R^N\times \mathbb R^N\mapsto \mathbb R$ is a positive semi definite kernel (i.e. kernelize linear regression)?


I'm assuming that by "linear regression" you mean unregularized linear regression, i.e. ordinary least squares. In that case, then yes, sure: this is just ridge regression with $\lambda = 0$. If the kernel matrix $K$ is invertible, then everything still works.

In fact, for many popular choices of kernel (such as the Gaussian RBF), the matrix $K$ is guaranteed to be invertible. But this is true only in exact arithmetic; in practice, the trailing eigenvalues of $K$ will be far too close to zero for computer arithmetic to handle them properly, and you'll get extreme numerical instability. You'll also be extremely close to severe multicollinearity, so that numerically you'll run into all the same problems there (which are often addressed by ridge regression).

  • $\begingroup$ Ok thanks but how do you suggest to proceed? Because if we set $\lambda=0 $ and do as in ridge regression we'll stumble through a $1/\lambda$... $\endgroup$ – Geppetto97 Jan 21 '19 at 20:33
  • $\begingroup$ If you derive as in (4)-(7) of your linked note, you don't need a $1/\lambda$. (The "alternative derivation" would need to be changed here, because there's no constraint to put in the Lagrangian.) $\endgroup$ – Dougal Jan 21 '19 at 20:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.