3
$\begingroup$

Is $||Y-X\beta||_2^2 + \lambda\beta^T K\beta$ , the standard loss-function in kernel ridge regression, or is it different? Also, is the gaussian kernel a standard choice used for the kernel, in practice? If not, which kernels are used more often than not? Also, is $\lambda$ the only parameter to be tuned via cross-validation or is the kernel parameter like $\sigma$ in a gaussian kernel, also tuned via cross validation in practice? Please confirm and/or correct my understanding of Kernel ridge regression!

$\endgroup$

2 Answers 2

1
$\begingroup$
  1. The standard loss function for kernel ridge regression is: $||Y-K\beta||_2^2 + \lambda\beta^T K\beta$. The equation in your question is missing the kernel matrix K in the $L_2$ error term.

  2. In practice, the Gaussian (a.k.a. RBF) and polynomial kernels are popular choices and could be a good starting point. However, the choice of kernel generally depends on the problem at hand. Sometimes it may be helpful to think of the kernel as a similarity metric for the input data vectors. You may need to experiment with different kernels to make an appropriate choice for the specific dataset.

  3. Yes, in addition to $\lambda$, you will need to determine the kernel parameters through cross-validation.

$\endgroup$
1
  • $\begingroup$ What speaks against optimizing the Kernel parameters via gradient based methods? $\endgroup$
    – bayerj
    Sep 16, 2013 at 13:00
1
$\begingroup$

I would disagree on your first point. The $L_2$ regularized model is $$ \parallel Y-K\beta \parallel_2^2 + \lambda \beta^T R \beta $$ where K is the known kernel matrix and $R$ is the regularization matrix. $K=R$ is only a good choice, when the gaussian kernel is used. For more information please see A. Smola, B. Schölkopf, On a Kernel-based Method for Pattern Recognition, Regression, Approximation, and Operator Inversion, 1997

@author, the discussion about "good kernels" is rather popular. See this post for example: What function could be a kernel?

However, there are ways to compute an optimized kernel based on your regularization idea. You should find some approaches presented at NIPS.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.