I would like to understand better the geometry underlying the Kernel trick with the Gaussian Kernel. In particular my question is:

  • How the Kernel trick can be interpreted geometrically, in particular for the case of the Gaussian Kernel? For example if kernelize linear SVM we would consider an hypothetical feature map $\Phi$ in some $R^k$ and draw linear boundaries on the embedded space. The set of all possible decision boundaries in the embedded space are hyperplanes. Consider an initial $2D$ (two dimensional) feature space. How can we get a characterization/intution of how the set of possible decision boundaries looks like in the original feature space? E.g. can we show simple things like if they will always be connected or not?

Some random thoughts. The Kernel trick with the Guassian Kernel amounts to the replacement:

$\Phi(x)\cdot \Phi(y) \rightarrow K(x,y)=\exp(|x-y|^2/2)$

Some consequences:

  1. if $x=y$ than this implies $|\Phi(x)|^2=\exp(0)=1$. So the feature map maps everything to a sphere. If the original space if 2 dimensional and $k=3$ (just to get some intution) I can assume that there are more hyperplanes dividing the data in the 2-dim sphere rather than in the original 2D plane, since the sphere has a curvature.

  2. Now if $\Phi$ is such a map we should have that the mapping induces a new distance in the original feature space $d(x,y)=||\Phi(x)-\Phi(y)||$(*). In Kernel terms $d(x,y)=\sqrt{K(x,x)+K(y,y)-2K(x,y)}$. For the Guassian Kernel this implies that $d(x,y)=\sqrt{2(1-\exp(|x-y|^2/2))}$ should be a distance. But this distance is quite weird and difficult to visualize since points very far apart initially get compressed to a finite distance of $\sqrt{2}$ (I guess this is a consequence of point 1).

*e.g. triangular inequality $d(x,z)=||\Phi(x)-\Phi(z)||\le ||\Phi(x)-\Phi(y)||+||\Phi(y)-\Phi(z)||=d(x,y)+d(y,z).$

  • $\begingroup$ Essentially, the RBF kernel projects all your points onto a segment of a hypersphere in a $N$-dimensional space. Maybe my answer here offers some intuition: stats.stackexchange.com/a/521569/169343 $\endgroup$
    – Igor F.
    Aug 10, 2023 at 7:59
  • $\begingroup$ It is interesting the observation that given $n$ points starting from example in $R^2$, $x_1,...,x_n$, you can find explicitely $n$ vectors, $\phi(x_1),...,\phi(x_n)$ in the sphear of an $n$ dimensional space with scalar products equal to the values given by the Kernel. Is this really the map underlying the RBF Kernel? Anyway this would define the feature map $\phi : R^2 ->R^n$ only at these selected points. To see the decision boundaries in the original space don't we need the map defined on the whole $R^2$? Maybe I am a bit confused... $\endgroup$
    – Thomas
    Aug 10, 2023 at 9:37


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.