Gaussian RBF vs. Gaussian kernel What is the difference between doing linear regression with a Gaussian Radial Basis Function (RBF) and doing linear regression with a Gaussian kernel?
 A: The only real difference is in the regularisation that is applied.  A regularised RBF network typically uses a penalty based on the squared norm of the weights.  For the kernel version, the penalty is typically on the squared norm of the weights of the linear model implicitly constructed in the feature space induced by the kernel.  The key practical difference this makes is that the penalty for the RBF network depends on the centers of the RBF network (and hence on the sample of data used) whereas for the RBF kernel, the induced feature space is the same regardless of the sample of data, so the penalty is a penalty on the function of the model, rather than on its parameterisation.
In other words, for both models we have
$f(\vec{x}') = \sum_{i=1}^\ell \alpha_i \mathcal{K}(\vec{x}_i, \vec{x}')$
For the RBF network approach, the training criterion is
$L = \sum_{i=1}^\ell (y_i - f(\vec{x}_i))^2 + \lambda \|\alpha\|^2$ 
For the RBF kernel method, we have that $\mathcal{K}(\vec{x},\vec{x}') = \phi(\vec{x})\cdot\phi(\vec{x}')$, and $\vec{w} = \sum_{i=1}^\ell \alpha_i\phi(\vec{x}_i)$.  This means that a squared norm penalty on the weights of the model in the induced feature space, $\vec{w}$ can be written in terms of the dual parameters, $\vec{\alpha}$ as
$\|\vec{w}\|^2 = \vec{\alpha}^T\matrix{K}\vec{\alpha},$
where $\matrix{K}$ is the matix of pair-wise evaluations of the kernel for all training patterns. The training criterion is then
$L = \sum_{i=1}^\ell (y_i - f(\vec{x}_i))^2 + \lambda \vec{\alpha}^T\matrix{K}\vec{\alpha}$. 
The only difference between the two models is the $\matrix{K}$ in the regularisation term.
The key theoretical advantage of the kernel approach is that it allows you to interpret a non-linear model as a linear model following a fixed non-linear transformation that doesn't depend on the sample of data.  Thus any statistical learning theory that exists for linear models automatically transfers to the non-linear version.  However, this all breaks down as soon as you try and tune the kernel parameters, at which point we are back to pretty much the same point theoretically speaking as we were with RBF (and MLP) neural networks.  So the theoretical advantage is perhaps not as great as we would like.
Is it likely to make any real difference in terms of performance?  Probably not much.  The "no free lunch" theorems suggest that there is no a-priori superiority of any algorithm over all others, and the difference in the regularisation is fairly subtle, so if in doubt try both and choose the best according to e.g. cross-validation.
