Loss for Kernel Ridge Regression

Question

Is $||Y-X\beta||_2^2 + \lambda\beta^T K\beta$ , the standard loss-function in kernel ridge regression, or is it different? Also, is the gaussian kernel a standard choice used for the kernel, in practice? If not, which kernels are used more often than not? Also, is $\lambda$ the only parameter to be tuned via cross-validation or is the kernel parameter like $\sigma$ in a gaussian kernel, also tuned via cross validation in practice? Please confirm and/or correct my understanding of Kernel ridge regression!

Answer 1

1. The standard loss function for kernel ridge regression is: $||Y-K\beta||_2^2 + \lambda\beta^T K\beta$. The equation in your question is missing the kernel matrix K in the $L_2$ error term.

2. In practice, the Gaussian (a.k.a. RBF) and polynomial kernels are popular choices and could be a good starting point. However, the choice of kernel generally depends on the problem at hand. Sometimes it may be helpful to think of the kernel as a similarity metric for the input data vectors. You may need to experiment with different kernels to make an appropriate choice for the specific dataset.

3. Yes, in addition to $\lambda$, you will need to determine the kernel parameters through cross-validation.

Answer 2

I would disagree on your first point. The $L_2$ regularized model is $$ \parallel Y-K\beta \parallel_2^2 + \lambda \beta^T R \beta $$ where K is the known kernel matrix and $R$ is the regularization matrix. $K=R$ is only a good choice, when the gaussian kernel is used. For more information please see A. Smola, B. Schölkopf, On a Kernel-based Method for Pattern Recognition, Regression, Approximation, and Operator Inversion, 1997

@author, the discussion about "good kernels" is rather popular. See this post for example: What function could be a kernel?

However, there are ways to compute an optimized kernel based on your regularization idea. You should find some approaches presented at NIPS.