I have been trying to understand the difference between a regular Support Vector Machine, and a kernel Support Vector Machine. I have my own intuition, but I'm not sure if it is quite right. Below is my understanding in my own words; is this correct?
So my understanding is that with a regular SVM, you are trying to find a hyperplane which separates the data into classes, based on their ground-truth labels. The problem involves optimising the parameters of the separating hyperplane, such that the distance between the hyperplane and the closest data point for each class, is maximum. In this context, the "distance" is calculated in the original space of the data.
However, if the data is not linearly-separable, then we need to find a new space in which the data is linearly-separable, so that the "distance" from before can be calculated in this new space. You could just introduce some arbitrary new dimensions, e.g. (x1 * x2), (x1 * x1), etc., but there is no guarantee that this new space is appropriate. So instead, you use a kernel to compute the new space, which takes into account the actual locations of the datapoints, rather than just the space as a whole.
A kernel takes in two datapoints, and computes a scalar value which effectively describes the "similarity" of the two datapoints. Then, for each datapoint, the kernel is applied between this datapoint and every single other datapoint in the dataset. This datapoint can now be described by a vector of "similarities", one element for each of the other datapoints. In this new space, it may now be easier to separate out the data, because the new space actually represents similarities between datapoints, whereas in the original space, it just represented the real-world data.
And the reason why this new space is considered to have potentially infinite dimensions, is that you can have one dimension of the new space for each datapoint in your data; so as your dataset approaches infinite size, the dimensionality of the new dataset also approaches infinity.
Am I thinking about this in the right way?
Specifically, I'm not sure about the following:
1) My understanding of why the kernel SVM space is better than just introducing arbitrary new dimensions (because the kernel method actually considers the relationships between datapoints).
2) My understanding of how the optimal separating hyperplane is calculated in the new space correct (by taking a datapoint and computing its similarity to every other datapoint, and then creating a vector with one similarity value for each of these other datapoints).
3) My understanding of why the new space can be thought of as having virtually infinite dimensions (because the dataset can have a virtually infinite number datapoints, and therefore a virtually infinite number of "similarities" in the vector which describes new space).