0
$\begingroup$

I am following Andrew Ng's lecture notes on SVM from CS229.

What I am having trouble understanding is the new objective function. $min_{γ,w,b} \frac{1}{2} ||w||^2 + C\sum_{i = 1}^{m}ξ_i$

From what I understand, if we increase C to have really high value, the objective function will be large, and to compensate for that w need to be small. Now we know that reducing w results in the larger margin and vice versa. Therefore, in this case the margin should be large, but it isn't. For high values of C, the margin is small and its a Hard margin classification.

Similarly, for smaller values of C, it becomes a soft margin classification.

I don't understand how the C influences the value of w and maybe I am missing something here. I'd really appreciate if someone can give me a clear and intuitive understanding of how w, C and the concept of margin are interrelated to each other.

Thanks.

$\endgroup$

1 Answer 1

2
$\begingroup$

The best way to understand the role of the hyperparameter $C$ is to think in term of robustness of your classifier. In short, $C$ is the hyperparameter that you will have to tune to find the optimal trade off between two competing objectives, which are 1) to make sure that your classifier performs well on your training data and 2) that it will also generalize well on unseen data.

Let's assume that you begin by training a "classical" SVM (i.e. without the regularization terms $C$ and $\xi_i$). If your training data set is well behaved (i.e. without outliers and linearly separable), then end result should look like this:

enter image description here

with the solid line being the classifier. However, if you train the same kind of SVM on a data set containing outliers, the result will be the following:

enter image description here

Here, the red point is an outlier, and using the unregularized objective function will lead to the classifier drawn as a solid line (instead of the expected classifier represented by a dashed line that we would have obtained, had we not considered the outlier). The classifier performs well on your training set (since all data points are correctly classified), but will poorly generalize on new data points (any new point falling into the blue area will be missclassified). This classifier is said to overfit or to lack robustness.

One way to avid this pitfall is to modify your objective function so as to allow some of your training points to be missclassified (i.e. to have a margin of $1-\xi_i$ instead of $1$). But we don't want too many of these points to be missclassified (otherwise the whole point of training a SVM to perform classification disappears). We thus have two competing objectives:

  1. On the one hand, like in the classical case, we want the $\xi_i$ not to be too high, to make sure the boundary is violated as little as possible. This is represented by adding the $\sum_i \xi_i$ term in the objective function;
  2. On the other hand, we also want to ensure the robustness of the classifier. The constraints are thus relaxed and become $y^{(i)}(w^T x^{(i)} + b) \geq 1-\xi_i$ instead of $y^{(i)}(w^T x^{(i)} + b) \geq 1$.

The hyperparameter $C$ determines how much you care about the respective objectives. If you set $C$ to be very high, then the only way to minimize your objective function will be to set all the $\xi_i$ to $0$. You are thus back to the original case in which the constraints are not relaxed. In contrary, if you care about the robustness of your classifier, you should set $C$ to a small value (to allow the $\xi_i$ to take whichever values they want).

Finding the optimal value for $C$ can be done by performing cross validation. If you obtain a very low training error, but a high test error (which is a telling sign of overfitting), you should consider increasing the value of $C$ and redo the whole tuning.

Even if you know a priori that your data set is linearly separable, adding the regularization terms $\xi_i$ is still a good idea, as it will allow your classifier to generalize well in case of outliers.

$\endgroup$
3
  • $\begingroup$ So let me try to understand with an example. Suppose $C$ is infinite. In that case, the $ξ_i$'s should be zero in order to minimize the objective, in doing so, the functional margin for those become $y^{(i)}(w^T x^{(i)} + b) = 1$ and we are taking the outliers into consideration. As a result, our margin has to let these outliers position on the gutters of the large margin, and so the road becomes so narrow to accomodate them. While, if $C = 0$, its as if the outliers don't exist, and we only build the road wrt the points who has f. margin = 1, thus a wider road. Am I right? $\endgroup$ Oct 20, 2020 at 17:05
  • 1
    $\begingroup$ To be exact : if $C$ becomes large (infinite $C$ is a degenerate case), then the $\xi_i$ should indeed be close to 0, and the classifier will indeed try to correctly classify the outliers. By doing so, the margin becomes very narrow (look at how close some points are from the solid line in the second picture). If $C$ becomes small ($C = 0$ is a degenerate case) then the classifier allows some outliers to be misclassified. Some points will thus have a functional margin smaller than 1. The 'road' is simply built so as to minimize the cost function, nothing else. $\endgroup$ Oct 20, 2020 at 17:28
  • $\begingroup$ Let us know if you have further questions or need more explanation. If this answer or any other one solved your issue, please mark it as accepted :) $\endgroup$ Dec 17, 2020 at 12:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.