5
$\begingroup$

In the soft margin SVM; can someone please give me the intuition of why a high value of the the penalty parameter $C$ causes the SVM to tend towards a hard margin SVM? I am failing to see the logic behind this.

Looking at the objective function of the soft margin SVM: $L_P=\frac{1}{2}||w||^2+C \cdot \Sigma_{i=1}^N{\xi}_i$, I can see that a small value of $C$ (such as one that tends to zero) would mean that we have $L_P=\frac{1}{2}||w||^2+0$, which is the same as the hard margin SVM. Unfortunately this is not the case--a high value of $C$ is what corresponds to a hard margin SVM. Can someone please give me a gentle insight into this concept?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ The $\ksi$ are the errors, so with a high C you give a high weight to errors and the objective function minimisés the errors, so you try not to make errors or you strive for a hard margin $\endgroup$
    – user83346
    Nov 8, 2015 at 10:45
  • $\begingroup$ @fcop Let me try to paraphrase your response. let me know if I am correct. When C is large, the term $C \cdot \Sigma_{i=1}^N{\xi}_i$ dominates $L_p$, so when I minimize $L_P$, I am largely minimizing the error term $C \cdot \Sigma_{i=1}^N{\xi}_i$. On the other when C is small, the term $\frac{1}{2}||w||^2$ dominates $L_P$, so when I minimize $L_P$, I am actually minimizing $\frac{1}{2}||w||^2$ meaning that the minimization does not have much effect on the other term which is already small? $\endgroup$
    – Minaj
    Nov 8, 2015 at 10:55
  • 1
    $\begingroup$ I am saying that, for a larger $C$, the term $C \sum \xi_i$ dominates more, so for a hughe $C$ you strive for zero errors (the $\xi_i$ measure the misclassification errors). Classification with no errors is ''hard margin'' ... $\endgroup$
    – user83346
    Nov 8, 2015 at 12:10

1 Answer 1

Reset to default
2
$\begingroup$

$ξ_i$ is like the distance you're going to allow the ith point to fall inside the margin. If it is 0, you're not allowing it in the margin. If it is positive you're allowing it to fall inside the margin somewhat. (And no $ξ_i$ are negative by definition.)

If you allow the ith point to fall in the margin, you must penalize the objective function for this - specifically by $Cξ_i$. If C is very large, $ξ_i$ will likely be set to 0 in the optimization process because the decrease in $L_P$ by letting the ith point fall in the margin (which lets $||w||$ become small) is overpowered by the increase in $L_P$ from letting $ξ_i$ get large.

So large C corresponds to hard margin.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks @Ben. Let me have a go at paraphrasing your answer to make sure I understand it. By letting the ith point to fall in the margin, you are basically reducing ||w||. The term $Cξ_i$ compensates for this difference. Because it is a product, $ξ_i$ is small when C is large. So, as $C$ tends to infinity, $ξ_i$ tends to zero, which implies a hard margin SVM ...Is this correct? $\endgroup$
    – Minaj
    Nov 11, 2015 at 0:19
  • $\begingroup$ Yes that's correct $\endgroup$
    – Ben
    Nov 11, 2015 at 4:18
  • $\begingroup$ @Ben Why $ξ_i$ is likely be set to 0 if $C$ is very large? $\endgroup$
    – Kaushal28
    Apr 1, 2019 at 12:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.