What is the difference between doing linear regression with a Gaussian Radial Basis Function (RBF) and doing linear regression with a Gaussian kernel?

  • $\begingroup$ Welcome to the site, @user35965. Please spell out your acronyms. By "RBF", do you mean radial basis function? $\endgroup$ – gung - Reinstate Monica Dec 10 '13 at 14:21
  • 2
    $\begingroup$ Yes that's exatcty what I meant. Duly noted for future reference. $\endgroup$ – user35965 Dec 10 '13 at 16:20

The only real difference is in the regularisation that is applied. A regularised RBF network typically uses a penalty based on the squared norm of the weights. For the kernel version, the penalty is typically on the squared norm of the weights of the linear model implicitly constructed in the feature space induced by the kernel. The key practical difference this makes is that the penalty for the RBF network depends on the centers of the RBF network (and hence on the sample of data used) whereas for the RBF kernel, the induced feature space is the same regardless of the sample of data, so the penalty is a penalty on the function of the model, rather than on its parameterisation.

In other words, for both models we have

$f(\vec{x}') = \sum_{i=1}^\ell \alpha_i \mathcal{K}(\vec{x}_i, \vec{x}')$

For the RBF network approach, the training criterion is

$L = \sum_{i=1}^\ell (y_i - f(\vec{x}_i))^2 + \lambda \|\alpha\|^2$

For the RBF kernel method, we have that $\mathcal{K}(\vec{x},\vec{x}') = \phi(\vec{x})\cdot\phi(\vec{x}')$, and $\vec{w} = \sum_{i=1}^\ell \alpha_i\phi(\vec{x}_i)$. This means that a squared norm penalty on the weights of the model in the induced feature space, $\vec{w}$ can be written in terms of the dual parameters, $\vec{\alpha}$ as

$\|\vec{w}\|^2 = \vec{\alpha}^T\matrix{K}\vec{\alpha},$

where $\matrix{K}$ is the matix of pair-wise evaluations of the kernel for all training patterns. The training criterion is then

$L = \sum_{i=1}^\ell (y_i - f(\vec{x}_i))^2 + \lambda \vec{\alpha}^T\matrix{K}\vec{\alpha}$.

The only difference between the two models is the $\matrix{K}$ in the regularisation term.

The key theoretical advantage of the kernel approach is that it allows you to interpret a non-linear model as a linear model following a fixed non-linear transformation that doesn't depend on the sample of data. Thus any statistical learning theory that exists for linear models automatically transfers to the non-linear version. However, this all breaks down as soon as you try and tune the kernel parameters, at which point we are back to pretty much the same point theoretically speaking as we were with RBF (and MLP) neural networks. So the theoretical advantage is perhaps not as great as we would like.

Is it likely to make any real difference in terms of performance? Probably not much. The "no free lunch" theorems suggest that there is no a-priori superiority of any algorithm over all others, and the difference in the regularisation is fairly subtle, so if in doubt try both and choose the best according to e.g. cross-validation.

  • 1
    $\begingroup$ @CagdasOzgenc Yes, for the RBF the regulariser is $\|\vec{\alpha}\|^2 = \vec{\alpha}^T\matrix{I}\vec{\alpha}$ rather than the $ \vec{\alpha}^T\matrix{K}\vec{\alpha}$ for the kernel machine. They would become more similar as the width of the basis function approaches zero as $K$ would approach $I$. I think this is essentially because $K$ is accounting for the correlation between basis functions. $\endgroup$ – Dikran Marsupial Oct 5 '17 at 6:45
  • $\begingroup$ @CagdasOzgenc The way I look at it is that the $K$ in the regulariser weights the penalisation differently for each basis vector, and the penalty depends on the selection of the other basis vectors. This weight depends on their correlations, so if you pick a different sample, the weights change to compensate. The other way to look at it is that the model is defined in a feature space determined by $\phi(x)$, which doesn't depend on the choice of basis vectors (providing they span the space containing the data). $\endgroup$ – Dikran Marsupial Oct 5 '17 at 11:08
  • $\begingroup$ @CagdasOzgenc Sure we can transform the space of the basis functions by an eigen-decomposition of $K$ and regain a $\|\vec{\alpha}'\|^2$ style regulariser (indeed that is a useful trick in optimising the regularisation parameter - doi.org/10.1016/j.neunet.2007.05.005). However that transformation eliminates the dependency of the original choice of basis function. For the two things to be equal would require $\vec{\alpha}^T\matrix{K}\vec{\alpha} = \mu\vec{\alpha}^T\matrix{I}\vec{\alpha}$, which is not generally true (especially not for the RBF kernel). $\endgroup$ – Dikran Marsupial Oct 5 '17 at 11:28
  • $\begingroup$ @CagdasOzgenc no problem, most of the standard texts explain it through eigenfunctions of the kernel function, which makes my brain hurt as well! ;o) $\endgroup$ – Dikran Marsupial Oct 5 '17 at 11:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.