1
$\begingroup$

Suppose I have a series of labelled training inputs $(x_i, y_i)$, and a kernel function $k$ on the input domain, with a corresponding RKHS $H$. Now form the Gram matrix $A$, where $A_{ij}=k(x_i, x_j)$. If $A$ is positive-definite, then there is a unique solution to

$$A\alpha=y$$

where $y$ is the vector of training labels. If I solve that system, then the function

$$f(x)=\sum_i\alpha_ik(x,x_i)$$

is, if I understand correctly, the minimum RKHS norm $f$ in the RKHS $H$ such that $f(x_i)=y_i$ for all $i$, by the Representer Theorem.

This defines a supervised learning procedure, by which I mean an algorithm which given a labelled training set, produces a hypothesis function on the input domain. Does this learning procedure have a name? I don't think it's equivalent to a Support Vector Machine, which is usually what comes up when I try googling for something like this. Kernel method seems to mean something more general, and kernel regression seems to mean something different - I don't recognize any of the formulae in that article.

$\endgroup$
1
  • $\begingroup$ In its most general meaning the problem is called interpolation. Since you have random data the specific instance is called scattered data interpolation" or maybe Gaussian regression or Kriging. $\endgroup$ – g g Mar 23 at 16:58
1
$\begingroup$

It's kernel ridge regression with zero weight decay (zero regularisation, $\lambda=0$).

Denote by $\textbf{k}$ the vector whose components are $k_i = k(x, x_i)$. Then you can write your $f(\textbf{x})$ in a vector form:

$$ f(\textbf{x}) = \alpha^T \textbf{k} = \textbf{y}^T \textbf{A}^{-1} \textbf{k} $$

For comparison, kernel ridge regression is given by

$$ f(\textbf{x}) = \textbf{y}^T (\textbf{A} + \lambda \textbf{I}) ^{-1} \textbf{k} $$

For reference, see e.g. Slide 31 here, or p. 119 in Cristianini and Shawe-Taylor, "An Introduction to Support Vector Machines and other kernel-based machine learning methods".

$\endgroup$
1
  • $\begingroup$ I'm not sure this is entirely accurate. If we seek a function in the RKHS minimizing the regularized loss $$\lVert f(x) - y\rVert_2^2 + \lambda \lVert f\rVert^2_{RKHS}$$ then with $\lambda=0$, this simply isn't a well-defined problem in general, since many functions in the RKHS fit the training data. However, I believe it is the case that for each $\lambda > 0$ the problem is well-defined, and if you let $\lambda\to 0$, the functions obtained converge pointwise to the function resulting from the procedure I described in the question. $\endgroup$ – Jack M Apr 29 at 14:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.