2
$\begingroup$

It is well know that the "usual" Radial Basis Function can be derived from Regularization that imposes small derivates. More precisely it is well known that the following:

$$ f(x) = \sum^{N}_{n=1} w_n \phi( \| x - x_n\| ) = \sum^{N}_{n=1} w_n e^{ - \gamma \| x - \mu_k \|^2 } $$

can be derived from the variational problem:

$$ H[h] = \sum^{N}_{n=1} (y_n - h(x_n))^2 + \lambda \sum^{\infty}_{k=0} \int^{\infty}_{- \infty} \left( \frac{d^k h }{dx^k} \right)^2 = \sum^{N}_{n=1} Loss(h(x_n), y_n) + \lambda R(h) $$

Where the nasty regularizer $R(h)$ is:

$$ R(h) = \sum^{\infty}_{k=0} \int^{\infty}_{- \infty} \left( \frac{d^k h }{dx^k} \right)^2 $$

However, one can consider a similar model where instead of considering all $N$ training points, we only consider $K$ (movable) centers $\{ \mu_1, ..., \mu_k, ... \mu_K \}$ (these movable centers can be estimated using k-means, for example).

Considering this alternative model:

$$ h(x) = \sum^{K}_{k=1} w_k \phi( \| x - x_k\| ) = \sum^{K}_{k=1} w_k e^{ - \gamma \| x - \mu_k \|^2 } $$

Is there any way to derive it from the minimization (optimization) of the empirical risk with some regularizer? Or can it be derived rigorously using Tikhonov Regukarization?


Intuitively, its clear that its not surprising that we can write the original Tikhonov Regularization solutions some kind of kernel (since Gaussians are smooth and the regularizer involves an infinite set of derivatives, the solutions is not too surprising to involve a Gaussian Kernel). This is not to surprising because of the Representer Theorem. However, I just can't see how movable centers with $K < N$ can be justified from a rigorous derivation.


I guess my main question is, how does one justify rigorously to use centers that are not the data points exactly and what its relationship is to the original regularization with radial basis functions and all the data points (as indicated by the representer theorem), since that is the only solution to be justified theoretically.

$\endgroup$
3
$\begingroup$

There are two papers that discuss this topic in great detail, Chen, Haykin, On Different Facets of Regularization Theory Girosi et Regularization Theory and Neural Networks Architectures

The answer is yes, you can rigorously derive it within the framework of Tikhonov regularization. The link between the regularizer and the gaussians in the Green function that results from minimizing $H[h]$. This is derived in the appendix of the second reference.

In case you are not very familiar with the functional derivative, check it in this book. It has, IMHO, one of the most accessible introduction to the topic.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.