How to understand effect of RBF SVM

Question

How can I understand what the RBF Kernel in SVM does? I mean I understand the maths, but is there a way to get a feeling when this kernel will be useful?

Would results from kNN be related to SVM/RBF since the RBF contains vector distances?

Is there a way to get a feeling for the polynomial kernel? I know the higher the dimension, the wigglier it is. But I'd like to get an intuition what the kernels do rather than try all possible kernels and pick the most successful.

Answer 1

You can possibly start by looking at one of my answers here:
Non-linear SVM classification with RBF kernel

In that answer, I attempt to explain what a kernel function is attempting to do. Once you get a grasp of what it attempts to do, as a follow up, you can read my answer to a question on Quora : https://www.quora.com/Machine-Learning/Why-does-the-RBF-radial-basis-function-kernel-map-into-infinite-dimensional-space/answer/Arun-Iyer-1

Reproducing the content of the answer on Quora, in case you don't have a Quora account.

Question: Why does the RBF (radial basis function) kernel map into infinite dimensional space? Answer : Consider the polynomial kernel of degree 2 defined by, $$k(x, y) = (x^Ty)^2$$ where $x, y \in \mathbb{R}^2$ and $x = (x_1, x_2), y = (y_1, y_2)$.

Thereby, the kernel function can be written as, $$k(x, y) = (x_1y_1 + x_2y_2)^2 = x_{1}^2y_{1}^2 + 2x_1x_2y_1y_2 + x_{2}^2y_{2}^2$$ Now, let us try to come up with a feature map $\Phi$ such that the kernel function can be written as $k(x, y) = \Phi(x)^T\Phi(y)$.

Consider the following feature map, $$\Phi(x) = (x_1^2, \sqrt{2}x_1x_2, x_2^2)$$ Basically, this feature map is mapping the points in $\mathbb{R}^2$ to points in $\mathbb{R}^3$. Also, notice that, $$\Phi(x)^T\Phi(y) = x_1^2y_1^2 + 2x_1x_2y_1y_2 + x_2^2y_2^2$$ which is essentially our kernel function.

This means that our kernel function is actually computing the inner/dot product of points in $\mathbb{R}^3$. That is, it is implicitly mapping our points from $\mathbb{R}^2$ to $\mathbb{R}^3$.

Exercise Question : If your points are in $\mathbb{R}^n$, a polynomial kernel of degree 2 will map implicitly map it to some vector space F. What is the dimension of this vector space F? Hint: Everything I did above is a clue.

Now, coming to RBF.

Let us consider the RBF kernel again for points in $\mathbb{R}^2$. Then, the kernel can be written as $$k(x, y) = \exp(-\|x - y\|^2) = \exp(- (x_1 - y_1)^2 - (x_2 - y_2)^2)$$ $$= \exp(- x_1^2 + 2x_1y_1 - y_1^2 - x_2^2 + 2x_2y_2 - y_2^2) $$ $$ = \exp(-\|x\|^2) \exp(-\|y\|^2) \exp(2x^Ty)$$ (assuming gamma = 1). Using the taylor series you can write this as, $$k(x, y) = \exp(-\|x\|^2) \exp(-\|y\|^2) \sum_{n = 0}^{\infty} \frac{(2x^Ty)^n}{n!}$$ Now, if we were to come up with a feature map $\Phi$ just like we did for the polynomial kernel, you would realize that the feature map would map every point in our $\mathbb{R}^2$ to an infinite vector. Thus, RBF implicitly maps every point to an infinite dimensional space.

Exercise Question : Get the first few vector elements of the feature map for RBF for the above case?

Now, from the above answer, we can conclude something:

• It may be quite hard to predict in general what the mapping function $\Phi$ looks like for arbitrary kernel. Though, for some cases like polynomial and RBF we can see what it looks like.
• Even when we know the mapping function, the exact effect that kernel will have on our set of points may be hard to predict. However, in certain cases we can say some things. For example, look at the $\Phi$ map given above for degree 2 polynomial kernel for $\mathbb{R}^2$. It looks like $\Phi(x) = (x_1^2, \sqrt{2}x_1x_2, x_2^2)$. From this we can determine that this map collapses diametrically opposite quadrants i.e first and third quadrant are mapped to same set of points and second and fourth quadrant are mapped to the same set of points. Therefore, this kernel allows us to solve XOR problem! In general, however, it might be harder to predict such behaviour for multidimensional spaces. And it gets harder in the case of RBF kernels.