0
$\begingroup$

What are kernels in support vector machines? I have tried many contents but i am not familiar with Lagrange and Laplace concept in mathematics. So anyone can please elaborate concepts of kernels in easy way?

Thanks in advance :)

$\endgroup$
1
$\begingroup$

In principle, a Kernel is just a feature transformation in an (infinite) feature space. It is often the case, that your feature space is to simple/small, so that you are not able to divide the data properly (in a linear way). Just look at the pciture of this blog (https://towardsdatascience.com/understanding-the-kernel-trick-e0bc6112ef78): In an 2D Feature space, you have no chance to separte datapoints in a linear way. Therefore just use a transformation (gaussian-kernel, polynomial kernel, etc. ) to achieve a higher feature space. In the 3D space, the circle can be divided by a linear function.

enter image description here

In principal, those kernels are just a function, which is computed on every datapoint. The mathematical trick behind those kernels is, that you do not have to actually compute this transformation on each datapoint. But this goes to far i think.

$\endgroup$
  • 2
    $\begingroup$ A kernel is NOT a mapping into feature space. It's a function that computes inner products in feature space. One could say that the choice of kernel implicitly determines a feature space mapping, but this is different than the kernel itself being such a mapping. $\endgroup$ – user20160 Aug 20 '19 at 15:57
  • $\begingroup$ Whats that linear and non-linear thing in this setting? Does it matters that hyerplane is linear or non-linear? $\endgroup$ – Yash Patel Aug 21 '19 at 10:32
  • $\begingroup$ user20160: this is right, I was to sloppy here. $\endgroup$ – TBockmair Aug 22 '19 at 6:43
  • $\begingroup$ Yash Patel: A SVM is in principle a linear classifier. In easy words, a SVM tries to separate datapoints in a linear way, hence with a linear decision function. It is often shown as a line trough datapoints. But often data is not separable by a simple line. Therefore we use Kernels. We expand the feature space, for example 2dimensional (feature1, feature2) to a much higher feature space. This transition from lower feature space to higher feature space can be calculated with more or less no costs with the Kernel Trick (inner products). $\endgroup$ – TBockmair Aug 22 '19 at 6:53
  • $\begingroup$ Does it matters that hyerplane is linear or non-linear? I think with hyperplane you mean the actual decision boundary of the SVM. The decision boundary is always linear, at least in the transformed feature space. In pictures you often see a wiggling line which separates the datapoints (the decision boundary, looks not linear). But in fact, the actual classification took place in the much higher dimensional feature space, where the decision boundary can be seen as linear. $\endgroup$ – TBockmair Aug 22 '19 at 7:01
1
$\begingroup$

We define kernels as real-valued functions $\kappa(x,x')\in\mathbb{R}$ where $x,x'\in\mathbb{R}^n$.

Typically,

  • $\kappa(x,x')\geq 0$
  • $\kappa(x,x')=\kappa(x',x)$

So a kernel can be interpreted as a measure of similarity. For example, $$\kappa(x,x')=x^Tx'$$

What we use in support vector machines are Mercer kernels. If a kernel is Mercer, then there exists a function $\phi:\mathbb{R}^n\rightarrow\mathbb{R}^m$ for some $m$ (which can also be infinite as in the case of the RBF kernel), such that:

$$\kappa(x,x')=\phi(x)^T\phi(x')$$

For example, let $\kappa(x,x') = (1+x^Tx')^2$ for $x,x'\in\mathbb{R}^2$.

$\Rightarrow\kappa(x,x') = (1+x_1x'_1+x_2x'_2)^2$

$\Rightarrow\kappa(x,x') = 1+2x_1x'_1+2x_2x'_2+(x_1x'_1)^2+(x_2x'_2)^2+2x_1x'_1x_2x'_2$

$\kappa(x,x')$ can be written as $\phi(x)^T\phi(x')$ where $\phi(x) = [1,\sqrt{2}x_1,\sqrt{2}x_2,x_1^2,x_2^2,\sqrt{2}x_1x_2]^T$.

So this kernel is equivalent to working in a 6-dimensional space.

Also, the complexity to get $(1+x^Tx')^2$ for $x,x'\in\mathbb{R}^2$ is lesser than the complexity to get $\phi(x)^T\phi(x')$ for $\phi(x),\phi(x')\in\mathbb{R}^6$.

$\endgroup$
  • $\begingroup$ Note: We are not using this particular setup. For example, for the RBF kernel this transformation $\Phi$ does not map into a finite dimensional space but into the infinite dimensional space $l^2$. What you are referring to is then $k(x,x') = \langle \Phi(x), \Phi(x')\rangle$, i.e. a kernel is almost a dot product (up to this weird and often complicated transformation $\Phi$). $\endgroup$ – Fabian Werner Aug 20 '19 at 9:56
-2
$\begingroup$

Try to read yourself at https://data-flair.training/blogs/svm-kernel-functions/

In a nutshell, they are various types of function that translate your input into a range between [0, 1] (both inclusive).

$\endgroup$
  • 1
    $\begingroup$ I've downvoted because kernel functions are not restricted to $[0,1]$. $\endgroup$ – Sycorax says Reinstate Monica Aug 20 '19 at 15:36

Not the answer you're looking for? Browse other questions tagged or ask your own question.