96
$\begingroup$

When using SVM, we need to select a kernel.

I wonder how to select a kernel. Any criteria on kernel selection?

$\endgroup$
  • 1
    $\begingroup$ what is the size of the problem? (#variables, observations)? $\endgroup$ – user603 Nov 7 '11 at 11:26
  • 1
    $\begingroup$ I am just asking for a generalized solution, no particular problem specified $\endgroup$ – xiaohan2012 Nov 7 '11 at 11:37
55
$\begingroup$

The kernel is effectively a similarity measure, so choosing a kernel according to prior knowledge of invariances as suggested by Robin (+1) is a good idea.

In the absence of expert knowledge, the Radial Basis Function kernel makes a good default kernel (once you have established it is a problem requiring a non-linear model).

The choice of the kernel and kernel/regularisation parameters can be automated by optimising a cross-valdiation based model selection (or use the radius-margin or span bounds). The simplest thing to do is to minimise a continuous model selection criterion using the Nelder-Mead simplex method, which doesn't require gradient calculation and works well for sensible numbers of hyper-parameters. If you have more than a few hyper-parameters to tune, automated model selection is likely to result in severe over-fitting, due to the variance of the model selection criterion. It is possible to use gradient based optimization, but the performance gain is not usually worth the effort of coding it up).

Automated choice of kernels and kernel/regularization parameters is a tricky issue, as it is very easy to overfit the model selection criterion (typically cross-validation based), and you can end up with a worse model than you started with. Automated model selection also can bias performance evaluation, so make sure your performance evaluation evaluates the whole process of fitting the model (training and model selection), for details, see

G. C. Cawley and N. L. C. Talbot, Preventing over-fitting in model selection via Bayesian regularisation of the hyper-parameters, Journal of Machine Learning Research, volume 8, pages 841-861, April 2007. (pdf)

and

G. C. Cawley and N. L. C. Talbot, Over-fitting in model selection and subsequent selection bias in performance evaluation, Journal of Machine Learning Research, vol. 11, pp. 2079-2107, July 2010.(pdf)

$\endgroup$
  • 1
    $\begingroup$ I get how the radial basis is a similarity measure since it's nearly 0 for vectors far away from each other and reaches its maximum on identical vectors. However, I don't see how that idea applies to the linear algorithm (using dot product as a kernel). How can we interpret the dot product as a similarity measure? $\endgroup$ – Bananin Mar 19 at 2:22
  • 1
    $\begingroup$ @Bananin the dot product can be written as the product of the magnitude of the two vectors times the cosine of the angle between them, so you can think of it as measuring the similarity in terms of the direction of the vectors (but obviously also dependent on their magnitudes) $\endgroup$ – Dikran Marsupial Mar 19 at 13:41
35
$\begingroup$

If you are not sure what would be best you can use automatic techniques of selection (e.g. cross validation, ... ). In this case you can even use a combination of classifiers (if your problem is classification) obtained with different kernel.

However, the "advantage" of working with a kernel is that you change the usual "Euclidean" geometry so that it fits your own problem. Also, you should really try to understand what is the interest of a kernel for your problem, what is particular to the geometry of your problem. This can include:

  • Invariance: if there is a familly of transformations that do not change your problem fundamentally, the kernel should reflect that. Invariance by rotation is contained in the gaussian kernel, but you can think of a lot of other things: translation, homothetie, any group representation, ....
  • What is a good separator ? if you have an idea of what a good separator is (i.e. a good classification rule) in your classification problem, this should be included in the choice of kernel. Remmeber that SVM will give you classifiers of the form

$$ \hat{f}(x)=\sum_{i=1}^n \lambda_i K(x,x_i)$$

If you know that a linear separator would be a good one, then you can use Kernel that gives affine functions (i.e. $K(x,x_i)=\langle x,A x_i\rangle+c$). If you think smooth boundaries much in the spirit of smooth KNN would be better, then you can take a gaussian kernel...

$\endgroup$
  • $\begingroup$ In your answer, you mentioned that "The "advantage" of working with a kernel is that you change the usual "Euclidian" geometry so that it fits your own problem. Also, you should really try to understand what is the interest of a kernel for your problem, what is particular to the geometry of your problem." Can you give a few references to start with. Thanks. $\endgroup$ – Raihana May 12 '12 at 8:57
12
$\begingroup$

I always have the feeling that any hyper parameter selection for SVMs is done via cross validation in combination with grid search.

$\endgroup$
  • 2
    $\begingroup$ I have the same feeling $\endgroup$ – xiaohan2012 Nov 7 '11 at 13:09
  • 2
    $\begingroup$ grid search is a bad idea, you spend a lot of time searching in areas where performance is bad. Use gradient free optimisation algorithms, such as the Nelder-Mead simplex method, which is far more efficient in practice (e.g. fminsearch() in MATLAB). $\endgroup$ – Dikran Marsupial Nov 7 '11 at 13:14
  • $\begingroup$ No, use graphical models or Gaussian processes for global optimization in combination with expected information. (See 'Algorithms for hyper parameter optimization', Bergstra et al, forthcoming NIPS) $\endgroup$ – bayerj Nov 7 '11 at 15:11
3
$\begingroup$

In general, the RBF kernel is a reasonable rst choice.Furthermore,the linear kernel is a special case of RBF,In particular,when the number of features is very large, one may just use the linear kernel.

$\endgroup$
  • 2
    $\begingroup$ It depends on whether your data is linearly separable or not, not how many dimensions you have. When the number of features is very large, (again depending on the data), you'd apply dimensionality reduction first, PCA or LDA (linear or nonlinear kernel variants) $\endgroup$ – user39663 Oct 21 '14 at 14:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.