What are kernels in support vector machines? I have tried many contents but i am not familiar with Lagrange and Laplace concept in mathematics. So anyone can please elaborate concepts of kernels in easy way?
Thanks in advance :)
$\begingroup$
$\endgroup$
1
-
1$\begingroup$ Possible duplicate of How to intuitively explain what a kernel is? or stats.stackexchange.com/questions/3947/… or stats.stackexchange.com/questions/18030/… $\endgroup$– Sycorax ♦Commented Aug 20, 2019 at 15:35
Add a comment
|
3 Answers
$\begingroup$
$\endgroup$
5
In principle, a Kernel is just a feature transformation in an (infinite) feature space. It is often the case, that your feature space is to simple/small, so that you are not able to divide the data properly (in a linear way). Just look at the pciture of this blog (https://towardsdatascience.com/understanding-the-kernel-trick-e0bc6112ef78): In an 2D Feature space, you have no chance to separte datapoints in a linear way. Therefore just use a transformation (gaussian-kernel, polynomial kernel, etc. ) to achieve a higher feature space. In the 3D space, the circle can be divided by a linear function.
In principal, those kernels are just a function, which is computed on every datapoint. The mathematical trick behind those kernels is, that you do not have to actually compute this transformation on each datapoint. But this goes to far i think.
-
3$\begingroup$ A kernel is NOT a mapping into feature space. It's a function that computes inner products in feature space. One could say that the choice of kernel implicitly determines a feature space mapping, but this is different than the kernel itself being such a mapping. $\endgroup$ Commented Aug 20, 2019 at 15:57
-
$\begingroup$ Whats that linear and non-linear thing in this setting? Does it matters that hyerplane is linear or non-linear? $\endgroup$ Commented Aug 21, 2019 at 10:32
-
$\begingroup$ user20160: this is right, I was to sloppy here. $\endgroup$ Commented Aug 22, 2019 at 6:43
-
$\begingroup$ Yash Patel: A SVM is in principle a linear classifier. In easy words, a SVM tries to separate datapoints in a linear way, hence with a linear decision function. It is often shown as a line trough datapoints. But often data is not separable by a simple line. Therefore we use Kernels. We expand the feature space, for example 2dimensional (feature1, feature2) to a much higher feature space. This transition from lower feature space to higher feature space can be calculated with more or less no costs with the Kernel Trick (inner products). $\endgroup$ Commented Aug 22, 2019 at 6:53
-
$\begingroup$ Does it matters that hyerplane is linear or non-linear? I think with hyperplane you mean the actual decision boundary of the SVM. The decision boundary is always linear, at least in the transformed feature space. In pictures you often see a wiggling line which separates the datapoints (the decision boundary, looks not linear). But in fact, the actual classification took place in the much higher dimensional feature space, where the decision boundary can be seen as linear. $\endgroup$ Commented Aug 22, 2019 at 7:01
$\begingroup$
$\endgroup$
1
We define kernels as real-valued functions $\kappa(x,x')\in\mathbb{R}$ where $x,x'\in\mathbb{R}^n$.
Typically,
- $\kappa(x,x')\geq 0$
- $\kappa(x,x')=\kappa(x',x)$
So a kernel can be interpreted as a measure of similarity. For example, $$\kappa(x,x')=x^Tx'$$
What we use in support vector machines are Mercer kernels. If a kernel is Mercer, then there exists a function $\phi:\mathbb{R}^n\rightarrow\mathbb{R}^m$ for some $m$ (which can also be infinite as in the case of the RBF kernel), such that:
$$\kappa(x,x')=\phi(x)^T\phi(x')$$
For example, let $\kappa(x,x') = (1+x^Tx')^2$ for $x,x'\in\mathbb{R}^2$.
$\Rightarrow\kappa(x,x') = (1+x_1x'_1+x_2x'_2)^2$
$\Rightarrow\kappa(x,x') = 1+2x_1x'_1+2x_2x'_2+(x_1x'_1)^2+(x_2x'_2)^2+2x_1x'_1x_2x'_2$
$\kappa(x,x')$ can be written as $\phi(x)^T\phi(x')$ where $\phi(x) = [1,\sqrt{2}x_1,\sqrt{2}x_2,x_1^2,x_2^2,\sqrt{2}x_1x_2]^T$.
So this kernel is equivalent to working in a 6-dimensional space.
Also, the complexity to get $(1+x^Tx')^2$ for $x,x'\in\mathbb{R}^2$ is lesser than the complexity to get $\phi(x)^T\phi(x')$ for $\phi(x),\phi(x')\in\mathbb{R}^6$.
-
$\begingroup$ Note: We are not using this particular setup. For example, for the RBF kernel this transformation $\Phi$ does not map into a finite dimensional space but into the infinite dimensional space $l^2$. What you are referring to is then $k(x,x') = \langle \Phi(x), \Phi(x')\rangle$, i.e. a kernel is almost a dot product (up to this weird and often complicated transformation $\Phi$). $\endgroup$ Commented Aug 20, 2019 at 9:56
$\begingroup$
$\endgroup$
1
Try to read yourself at https://data-flair.training/blogs/svm-kernel-functions/
In a nutshell, they are various types of function that translate your input into a range between [0, 1] (both inclusive).
-
1$\begingroup$ I've downvoted because kernel functions are not restricted to $[0,1]$. $\endgroup$– Sycorax ♦Commented Aug 20, 2019 at 15:36