I actually understood the derivation behind support Vector Machine but I have a doubt about constraint equation. enter image description here

Why we have a constraint equation $\geq1$ if $y_i=1$ and $\leq-1$ if $y_i=-1$? Can we have any arbitrary constant instead of 1? If no, then what is rational behind having this particular value?

Any help is highly appreciated.


Yes, you can have any arbitrary, strictly positive constant instead of 1.

Why? First some background.

Math and separating hyperplane:

Support vector machines attempts to find a separating hyper-plane between sets $X$ and $Y$. Mathematically, the condition for a separating hyperplane is:

$$ \boldsymbol{w} \cdot \boldsymbol{x}_i - b < 0 \quad \quad \boldsymbol{w} \cdot \boldsymbol{y}_i - b > 0 $$

Observe that the inequalities are strict!

Numerical issues and practical solution:

Numerically, this formulation has practical problems. If the inequalities aren't strict, $\boldsymbol{w} = \boldsymbol{0}, b = 0$ is a trivial solution. Numerical optimization routines may give bizarre answers to this problem; standard floating point math isn't infinitely precise etc...

What to do? Let's replace the strict inequalities with non-strict inequalities plus some separation constant $t>0$: $$ \boldsymbol{w} \cdot \boldsymbol{x}_i - b \leq -t \quad \quad \boldsymbol{w} \cdot \boldsymbol{y}_i - b \geq t $$ Yay! Numerical optimization can handle this. Also observe that since $\boldsymbol{w}$ and $b$ are choices variables, the scale of $t$ really doesn't matter. It's totally arbitrary. So we can just make it simple for ourselves and choose 1. (You could even choose different positive values for both inequalities; it doesn't matter.)

$$ \boldsymbol{w} \cdot \boldsymbol{x}_i - b \leq -1 \quad \quad \boldsymbol{w} \cdot \boldsymbol{y}_i - b \geq 1 $$

Other interpretation:

As your text explains, another interepretation of this is that you're fitting two parallel hyperplanes, one touching the X set, one touching the Y set, with some distance between them.

  • $\begingroup$ Hi, Methew....Just wondering, can we have a asymmetrical constraint? $\endgroup$ – Vrutang May 24 '16 at 16:16
  • $\begingroup$ For an example, $w⋅x_i−b≤−t$ and $w⋅y_i−b≥d$ $\endgroup$ – Vrutang May 24 '16 at 16:17

In an SVM, $y\in\{-1,1\}$. This is because the classifier is measuring the distance from the separating hyperplane, so it's sufficient to just check the sign (last line of the image you posted). You can use any two distinct values, but this will only make your life harder because your derivation will involve lots of tedious arithmetic to move around constants.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.