How does linear SVMs function in multi dimensional feature space? I'm not able to picture how a linear SVM can perform classification in more than 2 dimensions. Also, when to chose linear SVMs and kernel based SVMs? For example, my dataset has over 5000 instances with 4000 features.


Linear SVM always works in the exact same way, even if you can't mentally wrap your head around the geometrics. Humans are generally bad at reasoning in more than three dimensions, so don't let that worry you.

Imagine linear SVM in one dimension. This would look something like this:

+ + + + + | - - - -

where | is your decision boundary. A hyperplane in one dimension is a cutoff value. In two dimensions, you get a line. In three dimensions, you get a plane, ...

With 4000 features in input space, you probably don't benefit enough by mapping to a higher dimensional feature space (= use a kernel) to make it worth the extra computational expense. Hence, use a linear kernel.

In fact, always use the linear kernel first and see if you get satisfactory results. Generally, you can try a nonlinear kernel if and only if you don't get good results with a linear kernel.

  • $\begingroup$ Thanks for your reply. I can imagine in three dimensions it would be a plane. What I don't understand is how would a linear SVM look in 4000 dimensional space. Sorry if the question is really naive. $\endgroup$ – user1213055 Jul 13 '15 at 12:59
  • 1
    $\begingroup$ It would be a hyperplane, which is the generalization of a plane to more dimensions. Suppose you fix 3997 out of your 4000 dimensions and look at the decision boundary in the remaining 3 free dimensions, you'd get a plane, regardless of which 3 you pick. Don't try to visualize the geometry in your mind, because our imagination is generally limited to what we can interpret spatially (i.e., in three dimensions or less). $\endgroup$ – Marc Claesen Jul 13 '15 at 13:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.