# What is it called when you find the best fit in an RKHS to some training data?

Suppose I have a series of labelled training inputs $$(x_i, y_i)$$, and a kernel function $$k$$ on the input domain, with a corresponding RKHS $$H$$. Now form the Gram matrix $$A$$, where $$A_{ij}=k(x_i, x_j)$$. If $$A$$ is positive-definite, then there is a unique solution to

$$A\alpha=y$$

where $$y$$ is the vector of training labels. If I solve that system, then the function

$$f(x)=\sum_i\alpha_ik(x,x_i)$$

is, if I understand correctly, the minimum RKHS norm $$f$$ in the RKHS $$H$$ such that $$f(x_i)=y_i$$ for all $$i$$, by the Representer Theorem.

This defines a supervised learning procedure, by which I mean an algorithm which given a labelled training set, produces a hypothesis function on the input domain. Does this learning procedure have a name? I don't think it's equivalent to a Support Vector Machine, which is usually what comes up when I try googling for something like this. Kernel method seems to mean something more general, and kernel regression seems to mean something different - I don't recognize any of the formulae in that article.

• In its most general meaning the problem is called interpolation. Since you have random data the specific instance is called scattered data interpolation" or maybe Gaussian regression or Kriging. – g g Mar 23 at 16:58

## 1 Answer

It's kernel ridge regression with zero weight decay (zero regularisation, $$\lambda=0$$).

Denote by $$\textbf{k}$$ the vector whose components are $$k_i = k(x, x_i)$$. Then you can write your $$f(\textbf{x})$$ in a vector form:

$$f(\textbf{x}) = \alpha^T \textbf{k} = \textbf{y}^T \textbf{A}^{-1} \textbf{k}$$

For comparison, kernel ridge regression is given by

$$f(\textbf{x}) = \textbf{y}^T (\textbf{A} + \lambda \textbf{I}) ^{-1} \textbf{k}$$

For reference, see e.g. Slide 31 here, or p. 119 in Cristianini and Shawe-Taylor, "An Introduction to Support Vector Machines and other kernel-based machine learning methods".

• I'm not sure this is entirely accurate. If we seek a function in the RKHS minimizing the regularized loss $$\lVert f(x) - y\rVert_2^2 + \lambda \lVert f\rVert^2_{RKHS}$$ then with $\lambda=0$, this simply isn't a well-defined problem in general, since many functions in the RKHS fit the training data. However, I believe it is the case that for each $\lambda > 0$ the problem is well-defined, and if you let $\lambda\to 0$, the functions obtained converge pointwise to the function resulting from the procedure I described in the question. – Jack M Apr 29 at 14:44