7
$\begingroup$

I have some questions about SVM:

  1. In SVM there is a nonlinear and linear SVM. What is the difference between them?

  2. To do classification in SVM, we will find the linearly separable boundary (hyperplane). And if we cannot find it, we must project our input space into a features space. What does that feature space mean?
    I read on the internet that some explain that the features space is a scalar, and we get that scalar by inner product. For example, I have an input space matrix $A=[\{(1,2); 0\},\{(0,6);1\}]$; where 1, 2, 0, 6 are the features and 0, 1 are the labels. How can I project my input space A into feature space?

  3. Why do most people say that with kernel tricks, we actually don't project our data into a higher space? But we project it into a low space (because of that kernel tricks = scalar)?

  4. There are many kernel types: linear, polynomial, RBF and sigmoid. How can we determine which kernel type we have to use for our data?

  5. On this webpage, we can see that with polynomial and RBF kernel, we don't find the linearly separable boundary / hyperplane, but there is a curve, a group (for RBF). But as mentioned before, SVM finds the linear hyperplane? Why are they different? I appreciate all answers.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
8
$\begingroup$
  1. Nonlinear SVM is a synonym for SVM with a kernel trick.
  2. The idea is that if there is no linear separation in the original space, it may exist in some other space, quite likely of a higher dimension. Kernel trick allows one to construct this space implicitly by messing with the dot product in the original space; that's why the result seems as there is a nonlinear boundary in the original space.
    The scalar stuff is just a way of interpreting the boundary hyperplane. You can map the points on an axis perpendicular to the boundary, hook the zero onto the intersection and then each object will get a single coordinate that is positive for one class and negative for the other.
  3. See above.
  4. Tough stuff; sometimes you know that the data will fit particular kernel, sometimes you hope that RBF will also be good enough this time but generally it is an extra hyperparameter to be tuned.
  5. See 2.
Improve this answer
$\endgroup$
6
  • $\begingroup$ If you cannot get a natural separation in the original space you might be able to get it in a higher dimensional space. The simplest example is using a polynomial to separate two classes. Say it is a cubic. Instead of fitting y=a+bx (a 2 dimensional space), you fit y=a+bx+cx^2+dx^3. But note that the latter can be thought of as 4 dimensional vector space with basis (1,x,x^2,x^3) and the fit is still linear in these vectors. So you have effectively embedded the 2 dimensional vector space in a 4 dimensional one and found a good linear separation in that space. $\endgroup$
    – Dave31415
    Mar 20, 2014 at 2:16
  • $\begingroup$ The kernel trick is all about not having to explicitly choose the dimension of the higher dimensional space or the embedding. You can just pick a kernel and a smoothing scale and it allows you to do the calculation more efficiently. Most SVM codes will choose the bandwidth automatically using cross validation. So you just pick a kernel. Often it is the case that many different kernels give similar separations. $\endgroup$
    – Dave31415
    Mar 20, 2014 at 2:19
  • $\begingroup$ @mbq: from the link that I gave (in 5th quest), so the separable hyperplane is not always linier?? It can be a curve or a circle to group each classes?? $\endgroup$
    – user3378327
    Mar 20, 2014 at 8:39
  • $\begingroup$ @Dave31415, But the dot product/ inner product can only be conducted in R^n space. So I will only be able to calculate 1xn or nx1 matrix. But how if I have 3x3 matrix?? $\endgroup$
    – user3378327
    Mar 20, 2014 at 8:42
  • $\begingroup$ No. That's the whole point of the kernel trick. The dot product in the high dimensional space can actually be calculated in the low dimensional space if you apply the non-linear kernel. $\endgroup$
    – Dave31415
    Mar 20, 2014 at 11:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.