I understand the basics of what a Support Vector Machines' aim is in terms of classifying an input set into several different classes, but what I don't understand is some of the nitty-gritty details. For starters, I'm a bit confused by the use of Slack Variables. What is their purpose?

I'm doing a classification problem where I've captured pressure readings from sensors I've placed on the insole of a shoe. A subject will sit, stand, and walk for a couple of minutes while pressure data is recorded. I want to train a classifier to be able to determine whether a person is sitting, standing or walking and be able to do that for any future test data. What classifier type do I need to try? What is the best way for me to train a classifier from the data I've captured? I have 1000 entries for sitting, standing and walking (3x1000=3000 total), and they all have the following feature vector form. (pressurefromsensor1, pressurefromsensor2, pressurefromsensor3, pressurefromsensor4)


1 Answer 1


I think you are trying to start from a bad end. What one should know about SVM to use it is just that this algorithm is finding a hyperplane in hyperspace of attributes that separates two classes best, where best means with biggest margin between classes (the knowledge how it is done is your enemy here, because it blurs the overall picture), as illustrated by a famous picture like this: alt text

Now, there are some problems left.
First of all, what to with those nasty outliers laying shamelessly in a center of cloud of points of a different class?
alt text
To this end we allow the optimizer to leave certain samples mislabelled, yet punish each of such examples. To avoid multiobjective opimization, penalties for mislabelled cases are merged with margin size with an use of additional parameter C which controls the balance among those aims.
Next, sometimes the problem is just not linear and no good hyperplane can be found. Here, we introduce kernel trick -- we just project the original, nonlinear space to a higher dimensional one with some nonlinear transformation, of course defined by a bunch of additional parameters, hoping that in the resulting space the problem will be suitable for a plain SVM:

alt text

Yet again, with some math and we can see that this whole transformation procedure can be elegantly hidden by modifying objective function by replacing dot product of objects with so-called kernel function.
Finally, this all works for 2 classes, and you have 3; what to do with it? Here we create 3 2-class classifiers (sitting -- no sitting, standing -- no standing, walking -- no walking) and in classification combine those with voting.

Ok, so problems seems solved, but we have to select kernel (here we consult with our intuition and pick RBF) and fit at least few parameters (C+kernel). And we must have overfit-safe objective function for it, for instance error approximation from cross-validation. So we leave computer working on that, go for a coffee, come back and see that there are some optimal parameters. Great! Now we just start nested cross-validation to have error approximation and voila.

This brief workflow is of course too simplified to be fully correct, but shows reasons why I think you should first try with random forest, which is almost parameter-independent, natively multiclass, provides unbiased error estimate and perform almost as good as well fitted SVMs.

  • 6
    $\begingroup$ (+1) Great that you add some pictures to illustrate the whole thing! $\endgroup$
    – chl
    Oct 24, 2010 at 17:54
  • 3
    $\begingroup$ @mbq Since you seem very competent in SVM, let me ask you to clarify my doubt: once we found the best separating hyperplane, what do we use it for? We can define SVM as a method that, firstly, chooses the best hyperplane to correctly classify data points, and, secondly, it uses this hyperplane to sever new data points in the two classes. Right? (I've some doubts on the second part) $\endgroup$ Apr 27, 2012 at 16:08
  • 4
    $\begingroup$ @DavideChicco.it Basically the core problem of the whole ML is to make a model that will reliably predict new data, so yes. $\endgroup$
    – user88
    Jun 21, 2012 at 22:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.