Can someone please tell me the difference between the kernels in SVM:

  1. Linear
  2. Polynomial
  3. Gaussian (RBF)
  4. Sigmoid

Because as we know that kernel is used to mapped our input space into high dimensionality feature space. And in that feature space, we find the linearly separable boundary..

When are they are used (under what condition) and why?

up vote 4 down vote accepted

The linear kernel is what you would expect, a linear model. I believe that the polynomial kernel is similar, but the boundary is of some defined but arbitrary order

(e.g. order 3: $ a= b_1 + b_2 \cdot X + b_3 \cdot X^2 + b_4 \cdot X^3$).

RBF uses normal curves around the data points, and sums these so that the decision boundary can be defined by a type of topology condition such as curves where the sum is above a value of 0.5. (see this picture )

I am not certain what the sigmoid kernel is, unless it is similar to the logistic regression model where a logistic function is used to define curves according to where the logistic value is greater than some value (modeling probability), such as 0.5 like the normal case.

  • So, we can get the linierly separable hyperplane (boundary),if we use the linear kernel?? and if we use polynomial or RBF kernel, the (for polynomial)hyperlane could be a circle of grouped classes (for RBF) and curve ?? is that right??scikit-learn.org/stable/modules/svm.html – user3378327 Mar 21 '14 at 4:33
  • Each of the kernals work for high dimension versions of their respective boundaries. Does that answer your question? You are not limited to three dimensions for any kernel that I am aware of. – John Yetter Mar 21 '14 at 4:37
  • I just wanna make it clear. So the boundary by using linear kernel is a linear? For RBF is like a circle of grouped class?? and for polynomial, it can be curve based on the degree of the polynomial?? – user3378327 Mar 21 '14 at 4:48
  • I would not say RBF is a circle of grouped class. My understanding is that it applies a function based upon a normal distribution at each data point, and sums these functions. Then a boundary is formed by the curve representing a certain value on that function. If someone who has contributed to an SVM library could chime in, that might help. I think that your understanding of the other two kernels is correct. – John Yetter Mar 21 '14 at 5:01
  • U said that Linier Kernel is what I expected (to get linierly separable class) by using Kernel. and by using SVM classifier, we called it LINIER SVM. But how if we can get the linierly separable data without any kernel in SVM. What we call it?? Still Linier SVM or Non Linier SVM?? – user3378327 Mar 22 '14 at 9:24

Relying on basic knowledge of reader about kernels.

Linear Kernel: $K(X, Y) = X^T Y$

Polynomial kernel: $K(X, Y) = (γ\cdot X^T Y + r)^d , γ > 0$

Radial basis function (RBF) Kernel: $K(X, Y) = \exp(\|X-Y\|^2/2σ^2)$ which in simple form can be written as $\exp(-γ \cdot \|X - Y\|^2), γ > 0$

Sigmoid Kernel: $K(X, Y) = \tanh(γ\cdot X^TY + r) $ which is similar to the sigmoid function in logistic regression.

Here $r$, $d$, and $γ$ are kernel parameters.

  • 2
    While the information in your answer is correct, I don't think it answers the question here raised, which is more towards what's the practical difference between them, i.e. when to use one or the other. – Firebug Oct 29 '16 at 19:25
  • Amazingly these simple definitions are hard to come by. They should be the first thing presented when talking about differences of kernels, yet there is a wide-spread failure to state them. – cammil Sep 9 '17 at 16:22
  • Is there any official source for these? (I tested them and they seem correct, but I'd like to be able to cite them.) – Christian Eriksson May 15 at 18:16

This question can be answered from theoretical and practical point of view. From theoretical according to No-Free Lunch theorem states that there are no guarantees for one kernel to work better than the other. That is a-priori you never know nor you can find out which kernel will work better.

From practical point of view consult the following page:

How to select kernel for SVM?

While reflecting on what a kernel is "good for" or when it should be used, there are no hard and fast rules.

If you're classifier/regressor is performing well with a given kernel, it is appropriate, if not, consider changing to another.

Insight into how your kernel may perform, specifically if it is a classification model, might be gained by reviewing some visualisation examples, e.g. https://gist.github.com/WittmannF/60680723ed8dd0cb993051a7448f7805

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.