When do kernel based method perform better than the regular

Question

I am used with linear models. I can see rising use of kernel based method particularly in machine learning.

The following is an example Gaussian kernel using gausspr function from the R package kernlab. The function is an implementation of Gaussian processes for classification and regression.

require(kernlab)
x <- seq(-20,20,0.1)
y <- sin(x)/x + rnorm(401,sd=0.03)

# regression with gaussian processes
foo <- gausspr(x, y)
foo

# predict and plot
ytest <- predict(foo, x)
plot(x, y, type ="l")
lines(x, ytest, col="red")

The kernel methods available in this package include:

rbfdot Radial Basis kernel function "Gaussian"

polydot Polynomial kernel function

vanilladot Linear kernel function

tanhdot Hyperbolic tangent kernel function

laplacedot Laplacian kernel function

besseldot Bessel kernel function

anovadot ANOVA RBF kernel function

splinedot Spline kernel

My first question how do I know it is better to use kernel versions?

The second question is how do we select which kernel to use?

Answer 1

There is no easy answer to this question.

Typically you would start experimenting using the linear kernel function (to which I assume you refer as "regular"). If data is not linearly separable then there would be errors. So, if the performance is not satisfactory you would have to try some non-linear kernel functions.

The typical next choices are RBF and polynomial kernel of degrees 2 or 3. What is best depends on the geometry of the samples of your problems. The idea is that you try to find a kernel function that maps your samples in a higher dimensional space, in which samples become linearly separable. In practice, you have to try one by one the different options until you find something that works satisfactory for your problem. However, the more exotic kernels typically have some specific type of application or data in mind and unless you know you need them probably you can skip them.

There are some trade-offs you should have in mind as you increase complexity and start to explore the non-linear kernels:

• it becomes easier to overfit the data (you start fitting noise)
• the computational complexity increases (more time/memory requirements)
• you have to tune more hyper parameters (like gamma and degree for polynomial kernels)

Answer 2

Just some more thoughts to previous answer:

What is Kernel or Kernel Method ?

Kernel or Positive-definite kernel is a generalization of a positive-definite matrix.In linear algebra, a symmetric n × n real matrix M is said to be positive definite if zTMz is positive for every non-zero column vector z of n real numbers. Here zT denotes the transpose of z. kernel methods are a class of algorithms for pattern analysis, whose best known member is the support vector machine (SVM).

What is SVM ?

From wikipedia: "In machine learning, support vector machines (SVMs, also support vector networks) are supervised learning models with associated learning algorithms that analyze data and recognize patterns, used for classification and regression analysis. Given a set of training examples, each marked as belonging to one of two categories, an SVM training algorithm builds a model that assigns new examples into one category or the other, making it a non-probabilistic binary linear classifier. An SVM model is a representation of the examples as points in space, mapped so that the examples of the separate categories are divided by a clear gap that is as wide as possible. New examples are then mapped into that same space and predicted to belong to a category based on which side of the gap they fall on.

In addition to performing linear classification, SVMs can efficiently perform a non-linear classification using what is called the kernel trick, implicitly mapping their inputs into high-dimensional feature spaces."

Kernel methods owe their name to the use of kernel functions, which enable them to operate in a high-dimensional, implicit feature space without ever computing the coordinates of the data in that space, but rather by simply computing the inner products between the images of all pairs of data in the feature space. This operation is often computationally cheaper than the explicit computation of the coordinates. This approach is called the kernel trick.

Algorithms capable of operating with kernels include the kernel perceptron, support vector machines (SVM), Gaussian processes, principal components analysis (PCA), canonical correlation analysis, ridge regression, spectral clustering, linear adaptive filters and many others. Any linear model can be turned into a non-linear model by applying the "kernel trick" to the model: replacing its features (predictors) by a kernel function.

Radial basis function

The (Gaussian) radial basis function kernel, or RBF kernel, is a popular kernel function used in support vector machine classification.

Fisher kernel The Fisher kernel, named in honour of Sir Ronald Fisher, is a function that measures the similarity of two objects on the basis of sets of measurements for each object and a statistical model. In a classification procedure, the class for a new object (whose real class is unknown) can be estimated by minimising, across classes, an average of the Fisher kernel distance from the new object to each known member of the given class.The Fisher kernel is the kernel for a generative probabilistic model. As such, it constitutes a bridge between generative and probabilistic models of documents.

Polynomial kernel The polynomial kernel is a kernel function commonly used with support vector machines (SVMs) and other kernelized models, that represents the similarity of vectors (training samples) in a feature space over polynomials of the original variables, allowing learning of non-linear models.

Intuitively, the polynomial kernel looks not only at the given features of input samples to determine their similarity, but also combinations of these. In the context of regression analysis, such combinations are known as interaction features. The (implicit) feature space of a polynomial kernel is equivalent to that of polynomial regression, but without the combinatorial blowup in the number of parameters to be learned.

The RBF kernel is more popular in SVM classification than the polynomial kernel. The most common degree is d=2.

Thus the answer to question which kernel to use is: (1) Try simplest one first and then swith to more complex ones - as overfitting is danger (2) Learn from other worked examples, for similar data types. This may be quick solution, many times