For a linear SVM, the documentation tells me the formula is: $$\frac{1}{2}w^Tw + C\sum_{i=1}^l\xi_i.$$

Please explain to me in layman's terms what $w$ and $ξ$ represent. Is w the distance to the hyperplane, the weight of different variables, ... ?

Also, is there a difference with the formula $$f(x)=w^T x+b$$ or is it just a different notation?


Those two formulae are different things:

  • $\frac{1}{2} w^T w + C \sum \xi_i$ is one form of the objective function, the function which is minimized over $w$, $b$, and $\xi_i$ (subject to certain constraints, which are where $b$ comes in) to find the best SVM solution.

  • Once you've found the model (defined by $w$ and $b$), predictions on new data $x$ are done by finding their distance from the decision hyperplane, $f(x) = w^T x + b$.

$w$ and $b$ define the decision hyperplane, which separates positives from negatives, $\{ x \mid w^T x + b = 0 \}$. So $w$ is perpindicular to that hyperplane. $|w_j|$ is also the weight of the corresponding feature dimension: if $w_j = 0$, that feature is ignored, and if $|w_j|$ is high, that feature is important to the SVM's decision (assuming all the features are scaled similarly).

SVMs are trained by maximizing the margin, which is the amount of space between the decision boundary and the nearest example. If your problem isn't linearly separable, though, there is no perfect decision boundary and so there's no "hard-margin" SVM solution. This is why the "soft-margin" SVM was introduced, which allows some points to be on the wrong side of the margin.

$\xi_i$ is the slack variable defining how much on the wrong side the $i$th training example is. If $\xi_i = 0$, the point was classified correctly and by enough of a margin; if it's between 0 and 1, the point was classified correctly but by less of a margin than the SVM wanted; if it's more than 1, the point was classified incorrectly. ($\xi_i$ isn't allowed to be negative.) Points with $\xi_i > 0$, as well as those with $\xi_i = 0$ closest to the decision boundary, are known as support vectors because they "support" the margin. These are important in a kernel SVM because they're the only ones you need to worry about when predicting on new data.

$C$ is a parameter of the problem that defines how soft the margin should be. As $C \to \infty$, you get a hard SVM; if $C = 0$, the SVM doesn't care about getting the right answer at all and will just choose $w = 0$. In practice, you usually try a few different values of $C$ and see how they perform.

This picture (source) illustrates the different variables, though it's for a kernel SVM; just say $\phi(x) = x$ and it'll be a linear SVM.

  • 1
    $\begingroup$ 1 is correct. 2 not quite: the distance from a point $x$ to the separating hyperplane is $|w^T x - b| = |f(x)|$. Each feature has only a single associated weight $w_j \in \mathbb{R}$, which is something like the importance of that feature but not quite. $\endgroup$ – Danica Jul 21 '14 at 4:14
  • 1
    $\begingroup$ Sorry, you're right, support vectors can have $\xi = 0$ if they're right on the margin. In your picture, if you want to classify days based on 35 different features, then each data point would be a 35-dimensional vector corresponding to a row of the matrix, and $w$ would also be 35-dimensional. $\endgroup$ – Danica Jul 21 '14 at 13:11
  • 1
    $\begingroup$ @Glenn yes, it's a much shorter way of writing the same thing. :) $\endgroup$ – Danica Jul 21 '14 at 20:08
  • 1
    $\begingroup$ $2^{-10}$ is small but not absurd for $C$, and yes, if your feature values are small then $C$ will also be smaller. If your fit isn't good you could try a nonlinear kernel.... $\endgroup$ – Danica Jul 21 '14 at 22:13
  • 1
    $\begingroup$ You could try bootstrapping, but this is now to the point where you should ask a new question. :) $\endgroup$ – Danica Jul 22 '14 at 2:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.