5
$\begingroup$

I think that a formulation of SVM for points x with label y is :

$$ \begin{align} \arg\min_{\substack{u,w,b}} \frac{1}{2} \cdot |w|^2 + C \cdot \sum_{i} u_i \\ s.t.\ \ y_i\cdot (w \cdot x_i + b) &\geq 1-u_i \\\\ u_i \geq 0 \\ \end{align} $$

In that formulation, if we take C = 0, what prevents $u_i$ to go to infinity, so that the two constraints are always satisfied ?

$\endgroup$
3
  • $\begingroup$ my guess is that C has to be non zero, but I cant quite prove it. $\endgroup$
    – nicolas
    Apr 7 '13 at 17:01
  • $\begingroup$ If you set $C$ to zero, you are saying you want to totally ignore violation of the margin constraint. If you actually want to do that, you shouldn't be using the soft margin form in the first place. $\endgroup$
    – Glen_b
    Apr 8 '13 at 0:10
  • $\begingroup$ indeed. I see that if we set C to 0 we must remove the u_i, but I thought there'd be some deeper insight . $\endgroup$
    – nicolas
    Apr 8 '13 at 10:15
2
$\begingroup$

I don't think the $u_i$s will go to infinity. If $C$ is set to zero this effectively disables the inequality constraints, so the optimisation problem is just to minimise the squared norm of the weights, which has a trivial solution at $w = 0$. Substituting this into $y_i(w \cdot x_i + b) \geq 1 - u_i$ gives $y_i b \geq 1 - u_i$ which is satisfied by $u_i = 1 - y_i b$. This means there will be a solution where the $u_i$ are finite, provided a sensible choice is made for $b$. Whether the software actually finds this solution is another matter though; however as setting C=0 is effectively telling the SVM to completely ignore the training data, it is perhaps not too surprising that the programmer didn't consider this!

$\endgroup$
1
$\begingroup$

Actually, the slack variable $u_i$ means the tolerance of inconsistent labeling with $y_i$ by the linear function $(wx_i +b)$. The factor $C$ "adjusts" the weight of total tolerance. Therefore, setting $C$ to zero might be problematic in the case that the set $\{x_i,y_i\}$ is not linearly separable!

$\endgroup$
1
  • 1
    $\begingroup$ not really : the slack would just go to infinity. hence my trouble and dismay... $\endgroup$
    – nicolas
    Apr 7 '13 at 19:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.