4
$\begingroup$

It is common knowledge that all SVMS try to optimize some permutation of the following equation with the provided constraints:

SVM optimize

however, my question lies on the purpose of the constraint >= 1 and <= -1

I feel as if +1/-1 is a bit arbitrary, and feel as though its only purpose is act as two disjoint decision boundaries that can be adjusted separately, as opposed to a binary decision boundary such as (>0 positive, else negative) - which when adjusted, impacts both the positive decision boundary and the negative boundary at the same time, well because they are the same boundary

If this is true, then would it follow that all svms would work equally fine with the constraints +0.5/-0.5 or +500/-500? Since the only requirement we really need is number thresholds that give us two disjoint intervals that can be independently adjusted.

Is this idea correct?

$\endgroup$
5
$\begingroup$

You are right that the condition is a little bit arbitrary. It is a mathematical artifact to make things easier. But the reason is not because you would adjust them separately.

Your separating hyperplane has the form $w^Tx-b=0$. For simplicity you can think only at hard-margin classifier. What you want is to find a separating hyperplane with a maximum margin by using your data. For binary classification you will then want all positive instances to lie one side of the pane, and all negative instances on the other side of the pane. You do not want points to lie on the boundary when you learn.

So the two things I said about learning instances can be written as two inequalities which are the margins:

$$w^Tx_+ -b > 0$$ $$w^Tx_- -b < 0$$

but this form is useless since you know already that some points lies on margins and it would be useful to know which of them (they will actually became your support vectors). An alternate form would be then:

$$w^Tx_+ -b \ge \gamma$$ $$w^Tx_- -b \le -\gamma$$

for some positive $\gamma$. From the margin definitions you see that you actually do not find margins independently, because the hyperplane lies precisely in between them. But now the problem look complex since we introduced a new unknown $\gamma$, and we still have to solve the problem. The problem just looks complex, but is not. The new unknown $\gamma$ is not necessary. Note that actually get rid of this unknown if we divide everything by gamma. This is an equivalent form:

$$\frac{1}{\gamma}w^Tx_+ - \frac{1}{\gamma}b \ge 1$$ $$\frac{1}{\gamma}w^Tx_- -\frac{1}{\gamma}b \le -1$$

and the hyperplane equation will become: $\frac{1}{\gamma}w^Tx - \frac{1}{\gamma}b = 0$. It is the same hyperplane as $w^Tx-b=0$. For example in two real dimensions the line: $y = x -1$ is the same with $2y = 2x-2$ right?

As a conclusion without loosing generality we can denote $w^T \leftarrow \frac{1}{\gamma}w^T$ and $b \leftarrow \frac{1}{\gamma}b$ and you will have your equations.

$\endgroup$
  • $\begingroup$ Hi, could you maybe expand on this sentence "but this form is useless since you know already that some points lies on margins"? I feel like your explanation is the first I might actually understand, but I'm not quite there yet... $\endgroup$ – user3629892 Oct 12 '18 at 12:24
  • $\begingroup$ And why is the new unknown gamma not necessary? $\endgroup$ – user3629892 Oct 12 '18 at 12:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.