4
$\begingroup$

How does linear SVMs function in multi dimensional feature space? I'm not able to picture how a linear SVM can perform classification in more than 2 dimensions. Also, when to chose linear SVMs and kernel based SVMs? For example, my dataset has over 5000 instances with 4000 features.

$\endgroup$
9
$\begingroup$

Linear SVM always works in the exact same way, even if you can't mentally wrap your head around the geometrics. Humans are generally bad at reasoning in more than three dimensions, so don't let that worry you.

Imagine linear SVM in one dimension. This would look something like this:

+ + + + + | - - - -

where | is your decision boundary. A hyperplane in one dimension is a cutoff value. In two dimensions, you get a line. In three dimensions, you get a plane, ...

With 4000 features in input space, you probably don't benefit enough by mapping to a higher dimensional feature space (= use a kernel) to make it worth the extra computational expense. Hence, use a linear kernel.

In fact, always use the linear kernel first and see if you get satisfactory results. Generally, you can try a nonlinear kernel if and only if you don't get good results with a linear kernel.

| cite | improve this answer | |
$\endgroup$
  • $\begingroup$ Thanks for your reply. I can imagine in three dimensions it would be a plane. What I don't understand is how would a linear SVM look in 4000 dimensional space. Sorry if the question is really naive. $\endgroup$ – user1213055 Jul 13 '15 at 12:59
  • 1
    $\begingroup$ It would be a hyperplane, which is the generalization of a plane to more dimensions. Suppose you fix 3997 out of your 4000 dimensions and look at the decision boundary in the remaining 3 free dimensions, you'd get a plane, regardless of which 3 you pick. Don't try to visualize the geometry in your mind, because our imagination is generally limited to what we can interpret spatially (i.e., in three dimensions or less). $\endgroup$ – Marc Claesen Jul 13 '15 at 13:01
0
$\begingroup$

I was even stuck at same question for sometime, then I reffered some of the websites through which I was able yo get the intution.

If you are not able to visualise in mind consider this..

For 1 dimensional dataset a point will be suitable to discriminate between different classes.

Increase a dimension up for a 2 dimensional plane of data a single line may discriminate it with/without applying kernel.

Another increase will be analogous to having data in three dimensions but hyperplane or seperating plane in 2 dimension as we know a plane is 2d.

According to following Wikipedia article's starting lines you may develop the intution.

https://en.m.wikipedia.org/wiki/Hyperplane

The hyperplane is just one dimension less than data in order to separate the data points into multiple classes.

Then for 4000 feature space it will be nothing other than 3999 dimensional plane (plane in order to seperate) or simply collection of the points with 3999 dimensions in order to seperate the data points.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.