I was studying SVM ML alghorythm and I was wondering about solution for non-linear cases. As I understand it for know, SVM tries to find hyperplane or object in defined n-dimensional space, which separates datapoints according to their classes.
This object has n-1 dimensions, so if there are two features (feature space is 2-dimensional), then hyperplane is actually just line splitting this space.
However, in many cases it is not possible to separate data linearly, and this in case a projection to higher-dimensional space can sometimes help, because then the possibility that data will be linearly separable is higher.
Also, i read about radial basis function and RBF kernel. As i understand it, the purpose of kernel, is to make computation more efficient (here is the first place where i have doubts whether i understand it correctly). But i dont want to ask about kernel, the most important question is about the radial basis function itself.
From what i read it seems to me that this function uses some radius around each data point belonging to one of the classes, and then elevates this point to another dimension, while creating some margin around it, specified by that radius (size of that radius is called gamma). To define shape of the margin, some other function (for example Gaussian) is used.
I also tried to simulate it via some graphs:
First i created very simple dataset with one feature (position on x) and two classes (green and purple):
Because our feature space is 1-dimensional, the hyperplane splitting this space should be just a point (0-dimensional). As you can see, it is not possible to separate data that way - there is no point which could separate data into purple and green groups.
So i decided to elevate (or project?) datapoints of one class (if it would be projection of both, it would not made any change, am i right?) to second dimension - on y axis (red points):
Now I created Gaussian RBF (the blue line), using formula which i found on Wikipedia:
Here is formula which i used (x2 are red points, h is gamma parameter):
I have several questions what to do next/how to understand it:
1) How hyperplane (line in this case) is set now? Would not be better to use guassian function (the blue line) as a "separator" of what should belong to green/purple class?
2) If this "cheat" with projection to another space is used, isn't it completely messing the original purpose of support vector machines and leading to immediate overfitting? This seems like using target variable as a regular feature, which should be unacceptable (the only acceptable information is distribution on x axis, which may be similar with test dataset, and thus it can be important information)
3) Isn't this very similar to KNN classifier? Like KNN uses closest neighbors - position in space generated by features, this way - RBF - also tries to guess class using the position of sample in space generated by features)
4) The most important one I really suspect myself, that i completely misunderstood whole concept of RBF, because my intuition tells me that the way I did it is much more stupid than it should be, so please, dont get angry if whole concept of my understanding is wrong. Thats why i am asking here.